When I move job to latest company, it’s first time Iam using beanstalkd so I had a good experience implementing beanstalkd for that because the developer backend using beanstalkd for queue job , so I thought I’d write down how that went and what I learned.
Why Beanstalkd?
Beanstalk is a simple, fast work queue.
Its interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running time-consuming tasks asynchronously.
So when you put things on the queue (or “tube” as they seem to be called in Beanstalkd), they stay there until a worker comes along and processes each one successfully.
It will retry if the worker doesn’t indicate it completed successfully, and the worker can also “bury or destroy” the job.
the queue will just build up a bit of a backlog while it works through it all, nothing will be lost or missed. Also beanstalk can store job with a binlog, so if the server goes down or beanstalk crashes, it can pick up again with the queue intact.
But Iam using Beanstalkd like when I say in above my latest company using Beanstalkd so I must follow that 😀 .
Working with Beanstalkd
Iam using Docker for my env with ubuntu 14 so for install beanstalkd it’s very simple using apt-get install beanstalkd.
It has a selection of client libraries so you can use whatever works best for the application you want to use it with.
By default beanstalk comes up on port 11300 but you can change port if you want.
For change port you can edit file with vi /etc/default/beanstalkd
root@4660a4d1f763:~# cat /etc/default/beanstalkd
## Defaults for the beanstalkd init script, /etc/init.d/beanstalkd on
## Debian systems.
BEANSTALKD_LISTEN_ADDR=10.10.10.10
BEANSTALKD_LISTEN_PORT=11300
# You can use BEANSTALKD_EXTRA to pass additional options. See beanstalkd(1)
# for a list of the available options. Uncomment the following line for
# persistent job storage.
BEANSTALKD_EXTRA="-b /var/lib/beanstalkd"
To be clear, the main configuration necessary here is the addition of the beanstalkd configuration:
BEANSTALKD_LISTEN_ADDR is address for beanstalkd installed.
BEANSTALKD_LISTEN_PORT is port listen beanstalkd thats I say you can change port you want.
BEANSTALKD_EXTRA this for extra option when Iam using -b /var/lib/beanstalkd it’s mean job will be store in disk /var/lib/beanstalkd not in memory.
that’s solution if beanstalkd server crash or reboot will not gone, they still keep job in disk.
for start you can use like systemv deamon /etc/init.d/beanstalkd start and you cant telneting to there like telnet localhost 11300
How is your beanstalkd feeling today? You can ask it by typing stats – this gives quite a lot of output but will include how many jobs are waiting/in progress/failed.
In fact from this telnet prompt you can look at a bunch of things; here’s a handful of my favourite commands:
list-tubes – shows which tubes are available/in use.
list-tubes
OK 64
---
- default
stats-tube [tube] – get the stats for a single tube.
stats-tube cdn_upload
OK 270
---
name: cdn_upload
current-jobs-urgent: 0
current-jobs-ready: 34
current-jobs-reserved: 0
current-jobs-delayed: 0
current-jobs-buried: 0
total-jobs: 34
current-using: 0
current-watching: 0
current-waiting: 0
cmd-delete: 0
cmd-pause-tube: 0
pause: 0
pause-time-left: 0
peek-ready – shows the next job to be processed in the current tube.
peek-ready
NOT_FOUND
use [tube] – use a specific tube.
When you are done, type ^] + enter and quit to return to your prompt.
Using these commands you can check in with your queue if you need to, but mostly I find I just need to make sure it is running, and that the workers are running, and beyond that I don’t need to think about it too much! I have detailed logging on the workers so if there are any issues, the information that I need in there.
