I am able to send any number of messages to a particular queue when i am not running any other application.
But when I am running the application which is continuously reading the queue, then if I send more than 1000 messages to that queue then some messages are lost, For example, i send 2000 messages, sometime i get 800, sometimes 1200, and when I kept
the send application wait until the user exits, then also i received only 1500, still 500 are lost.
Please give me your suggestions how to solve it.
Please check the server logs and configuration. There are predefined queue limits, which are presumably being enforced. You can increase these queue limits. You can also use direct mode if you have only one consumer per queue.
See http://www.openamq.org/doc:user-3-advanced#toc25 for more details.
Thanks for the reply,
I used the direct mode, it improved little bit, but still I am facing the problem, I check ed the log files, i didn't find any clue. I also see the config file, where the limits are set correctly, they are at 50000 and 100000, the limit is far above 15000 messages, where I am getting the problem. When I send 15000, i am getting only 4000, all the messages are missing.
When i searched for previous problems of missing messages, they given a solution: Flush messages to avoid loosing messages, but i don't know api, how to flush messages, please provide me with the syntax to flush messages.
There is no "flush messages" concept in AMQP. But if you're losing messages like this there is something wrong in either your routing or consuming.
Can you reproduce this with fewer messages, and if so, with how few? 10? If so, run the server with —debug_route 1 and run the test, and look for dropped messages.
It is unable to write messages more than 10000, i can't able to reproduce the problem for less than 10000 messages. But the limits in amq_server.cfg file is 50000 and 100000.
Is this problem, due to exchanges? I am using same exchange "amq.direct" for reading from the queues from one application and writing to the queues using other application
This is the amq_server.cfg I am using.
OpenAMQ server primary configuration file
** DO NOT EDIT THIS FILE **
To define a custom configuration, especially for security data and
cluster configuration, edit "amq_server.cfg", or pass configuration options
on the command line.
Security data and cluster configuration, and queue and user group
profiles cannot be defined on the command-line or via the shell. This
file provides example sections that you can copy to custom.cfg.
Run "amq_server —help" for options help.
Syntax for message transfer agent (MTA) configuration
name = "exchange-name"
vhost = "virtual-host"
host = "localhost:5000"
login = "peering"
mode = "0 | 1 | 2 | 3 | 4" 0=disabled, 1=subscriber, 2=fwd-all, 3=fwd-else, 4=both
These are default queue profiles:
Valid actions are:
warn - send message to alert log
drop - drop the message
trim - remove old messages from queue
kill - kill the connection and queue
Note that if you define a drop/trim action, the queue will never
grow larger than the specified limit.
Please define limits in ascending order.
<queue_profile name = "private">
<limit name = "warn" value = "50000" />
<limit name = "trim" value = "100000" />
<queue_profile name = "shared">
<limit name = "warn" value = "50000" />
<limit name = "kill" value = "100000" />
<!— These are the default passwords —>
<security name = "plain">
<user name = "guest" password = "guest" group = "normal" />
<user name = "super" password = "super" group = "super" />
<user name = "peering" password = "peering" group = "super" />
OK. The config file looks fine. I assume you're not using any client-side configuration (wireapi.cfg?)
Could you provide some more detail about what each producer and consumer is doing, in terms of routing keys, queues, bindings?
What I am doing the consumer is continuously binding with a queue and checking if there is any message consume it and unbinding or else return.
The publisher binds with that queue and writes as many number of messages to a the queue, as requested.
Both publisher and consumer using queue name as the routing key, and using same exchange "amq.direct"
The approach you are using is probably the cause of the error. If there is no active binding, messages will just be dropped.
Here is how you should design the application:
- At start-up, create your queue with the auto-delete option and bind it to the exchange using the queue name. Consume from the queue, and then wait for messages.
- Process messages, and do not unbind the queue at any point.
- At shut-down, just close the connection: the queue and binding will be deleted automatically.
The publisher does not need to make any binding but just publish to the exchange.
For a single queue it is ok, but I have two publishers writing in two separate queues, should i use two exchanges? or i have to use publisher binding to the queues in the same exchange? Please give me a solution, i am not abe to choose between bindings and exhanges.
You would use one exchange and two different routing keys, one per queue.
You can use the default exchange for this:
queue.declare queue1 (or allow the server to name the queue) queue.declare queue2 ... basic.publish exchange="" routing-key="queue1"
If you want to use an explicit exchange, create one binding for each queue.
You can try this very easily in PAL, and look at the PAL examples for more help (see main page for links to the PAL examples archive)
Here are the possible reasons why your applications might lose messages:
- They could be published with routing keys that do not match any bindings. The server will drop such messages silently. If you use —debug_route, it will print a warning. If you publish the messages with the immediate option, they will be returned to the sender.
- They could be put into a queue that overflows. The server will warn when this happens.
- They could be consumed by the 'wrong' application, i.e. if applications A and B both consume from the same queue then A might think that some messages are missing. In fact they went to B.
- They could be successfully delivered but lost at the client side, due to programming errors in the application.
- They could be delivered to the client but not collected in time, especially if the client crashes before it has processed waiting messages. We use acknowledgements for guarding against this kind of situation.
- They could be dropped at the client side by the WireAPI layer, if you configured that to do so (setting a 'high water' mark).
As far as I know, there is no other way messages could be lost.
Another diagnostic tool you can use is the '—dump_state 1' server option, which will report exactly how many messages were received, and sent out. If you use this option and run the test, the output will help identify what is causing your problem.