<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi Alberto,<br>
<br>
On 09/09/2013 04:40 AM, Alberto Klocker wrote:<br>
</div>
<blockquote
cite="mid:CAP7CNSXaMMTXFRoDGLntZGMZSafCB-vrtvpsv8NACTLEMouB=Q@mail.gmail.com"
type="cite">
<div dir="ltr">
<pre style="color:rgb(0,0,0)">Hi,</pre>
<pre style="color:rgb(0,0,0)">We run tests on a 10 Gb network environment with Polygraph as one of the benchmarking tools but have found we are hitting 100% CPU usage on both the poly-client and poly-server which would indicate we need to spread the processes and systems out.</pre>
<pre style="color:rgb(0,0,0)">How well does the reporting engine handle log files from multiple sources? </pre>
<pre style="color:rgb(0,0,0)"><span style="font-family:arial">Say I were to script a system that launches two or three instances of the client and server, will the reporting engine be capable of combining the results into one report?</span>
</pre>
</div>
</blockquote>
All the polygraph reporting tools (like polygraph-lx,
polygraph-ltrace, polygraph-reporter) support multiple logs
aggregation. E.g. if you have logs from three client workers named
clt.1.log, clt.2.log, clt.3.log you can extract statistics from them
simply using <br>
$ polygraph-lx clt.1.log clt.2.log clt.3.log. Same for others.
However, there is a minor bug that prevents from ltrace tool
extracting logs from both server and client sides in one run: <a
class="moz-txt-link-freetext"
href="https://bugs.launchpad.net/polygraph/+bug/1204983">https://bugs.launchpad.net/polygraph/+bug/1204983</a>
<blockquote
cite="mid:CAP7CNSXaMMTXFRoDGLntZGMZSafCB-vrtvpsv8NACTLEMouB=Q@mail.gmail.com"
type="cite">
<div dir="ltr">
<pre style="color:rgb(0,0,0)"><span style="font-family:arial">Would it be better I write something that "merges" the results together?</span></pre>
</div>
</blockquote>
As a result of written above, you don't have to write anything.
<blockquote
cite="mid:CAP7CNSXaMMTXFRoDGLntZGMZSafCB-vrtvpsv8NACTLEMouB=Q@mail.gmail.com"
type="cite">
<div dir="ltr">
<pre style="color:rgb(0,0,0)">I'm curious to see what others have done to ramp Polygraph up above its single core limits.</pre>
<pre style="color:rgb(0,0,0)">We can get 10 Gb throughput with large files but my goal is to find maximum, stable concurrent connections.</pre>
<pre style="color:rgb(0,0,0)">Any help is much appreciated!</pre>
</div>
</blockquote>
Sure Alberto. There is a number of actions that could help you
increase polygraph performance:<br>
<br>
1) Run multiple instances of workers and bind them to different cpu
cores. E.g if you have 2 cpu cores on machine that runs
polygraph-client, run two clients like:<br>
taskset -c 0 polygraph-client --log clt.1.log ...<br>
taskset -c 1 polygraph-client --log clt.2.log ...<br>
<br>
2) Bind tune cpu affinity for NIC interrupts to cores different from
the ones that already occupied by polygraph workers (clients or
servers). E.g. if you have 4 cpu cores, consider to run three
polygraph-client instances bound to cores 0-2 and bind NIC
interrupts to core 3. This article should help you with SMP affinity
in linux: <a class="moz-txt-link-freetext"
href="http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux">http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux</a>
. There is no good formula to say what should be ratio between
workers' and NIC's cores, but top output could give you the clues.<br>
<br>
3) If you have 'virtual' (hyper-threading provided) cores, it's not
a good idea to bind polygraph workers to both 'real' and 'virtual'
instances of the same core.<br>
<br>
4) If you have enough RAM, consider writing test results on
in-memory FS like tmpfs (<a class="moz-txt-link-freetext"
href="https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt">https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt</a>).<br>
<br>
5) Tuning linux TCP stack for 10G networking is important part too.
I'd use this document as a guide:
<a class="moz-txt-link-freetext" href="http://landley.net/kdocs/ols/2009/ols2009-pages-169-184.pdf">http://landley.net/kdocs/ols/2009/ols2009-pages-169-184.pdf</a> .<br>
<br>
Best wishes,<br>
Pavel
<blockquote
cite="mid:CAP7CNSXaMMTXFRoDGLntZGMZSafCB-vrtvpsv8NACTLEMouB=Q@mail.gmail.com"
type="cite">
<div dir="ltr">
<pre style="color:rgb(0,0,0)">
</pre>
<pre style="color:rgb(0,0,0)">Kevin Xie1 <<a moz-do-not-send="true" href="http://www.web-polygraph.org/mailman/listinfo/users">Kevin_Xie1 at symantec.com</a>> writes:
><i> Thanks for your quick response, Dmitry.
</i>><i>
</i>><i> However, I don't understand how a single thread can simulate multiple
</i>><i> concurrent "users" submitting request to web servers. Is it that the
</i>><i> single thread iterates all open connections one by one (submitting a
</i>><i> request or processing response one at a time)?
</i>
Sort of, though details are more complex than that. If you are
interested, search for I/O multiplexing.
><i> bound by CPU since just 1 core is used at a time, or by slow
</i>><i> connections ..., do I miss something here, or my understanding is
</i>><i> fundamentally wrong?
</i>><i>
</i>
The fact that a single polygraph process can use only a single core may
be a limitation indeed. But moving each simulated agent to a separate
OS thread is not the solution.
You can run multiple Polygraph client or server processes on a single
system. But that makes it harder to manage the test and usually
requires some auxiliary scripts to start all processes.
It would be nice to have SMP support in Polygraph and we have some ideas
about it. But so far there has not been enough interest in such
project. Patches or sponsorships are welcome.
In any case, please do not make conclusions about Polygraph performance
from the fact that it is single-threaded. Polygraph is successfully
used for high performance tests in 10Gbit networks (though it may
require multiple Polygraph processes and/or systems).
Regards,
Dmitry
><i> Appreciated for any light shed!
</i>><i>
</i>><i> Kevin
</i>><i>
</i>><i> -----Original Message-----
</i>><i> From: Dmitry Kurochkin [mailto:<a moz-do-not-send="true" href="http://www.web-polygraph.org/mailman/listinfo/users">dmitry.kurochkin at measurement-factory.com</a>]
</i>><i> Sent: May-03-12 3:19 PM
</i>><i> To: Kevin Xie1
</i>><i> Cc: <a moz-do-not-send="true" href="http://www.web-polygraph.org/mailman/listinfo/users">users at web-polygraph.org</a>
</i>><i> Subject: Re: Is a client robot an OS thread?
</i>><i>
</i>><i> Hi Kevin.
</i>><i>
</i>><i> Kevin Xie1 <<a moz-do-not-send="true" href="http://www.web-polygraph.org/mailman/listinfo/users">Kevin_Xie1 at symantec.com</a>> writes:
</i>><i>
</i>>><i> Hi,
</i>>><i>
</i>>><i> Is a client robot a real OS thread? I'm using web polygrah in Linux,
</i>>><i> but I doesn't see multiple threads in the OS during a test with 500
</i>>><i> robots.
</i>>><i>
</i>><i>
</i>><i> Polygraph client and server processes are single-threaded. All Robot
</i>><i> and Server agents are running in a single thread.
</i>><i>
</i>><i> Regards,
</i>><i> Dmitry
</i>><i>
</i>>><i> Thanks,
</i>>><i> Kevin</i></pre>
<div>
<div dir="ltr"><span
style="background-color:rgb(255,255,255);border-collapse:separate;color:rgb(34,34,34);font-size:medium;font-family:Helvetica">
<div style="word-wrap:break-word"><span
style="border-collapse:separate">
<div style="word-wrap:break-word">
<span style="border-collapse:separate">
<div style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span
style="font-size:8pt;font-family:Verdana,sans-serif;color:rgb(54,95,145)"
lang="EN-US"><b>Alberto Klocker</b> | Software
Developer</span><span
style="font-size:8pt;font-family:Verdana,sans-serif;color:rgb(54,95,145)"> | </span><span
lang="EN-US"><a moz-do-not-send="true"
href="http://www.bluereef.com.au/"
style="color:blue" target="_blank"><span
style="font-size:8pt;font-family:Verdana,sans-serif;color:rgb(54,95,145);text-decoration:none"
lang="EN-AU">www.bluereef.com.au</span></a></span><span
style="font-size:8pt;color:rgb(54,95,145)"
lang="EN-US"> </span><span
style="font-size:8pt;color:rgb(54,95,145)"></span></div>
<div style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><u><span
style="font-size:8pt;color:rgb(54,95,145)">_________________________________________________________________</span></u></div>
</span></div>
</span></div>
</span><span
style="background-color:rgb(255,255,255);color:rgb(34,34,34);font-size:15px;font-family:Calibri,sans-serif"><span
style="font-size:8pt;font-family:Verdana,sans-serif;color:rgb(54,95,145)"
lang="EN-US"><b>D: </b><a moz-do-not-send="true"
value="+61398958020" style="color:rgb(17,85,204)">+61
3 9895 8006</a> | </span></span><span
style="background-color:rgb(255,255,255);border-collapse:separate;color:rgb(34,34,34);font-size:medium;font-family:Helvetica"><span
style="font-size:15px;font-family:Calibri,sans-serif"><b><span
style="font-size:8pt;font-family:Verdana,sans-serif;color:rgb(54,95,145)"
lang="EN-US">T</span></b></span><span
style="font-size:15px;font-family:Calibri,sans-serif"><span
style="font-size:8pt;font-family:Verdana,sans-serif;color:rgb(54,95,145)">: <a
moz-do-not-send="true" value="+61398988000"
style="color:rgb(17,85,204)">+61 3 9898 8000</a> </span></span><span
style="font-size:15px;font-family:Calibri,sans-serif"><span
style="font-size:8pt;font-family:Verdana,sans-serif;color:rgb(54,95,145)">| </span></span><a
moz-do-not-send="true" value="+61398988000"
style="font-family:Verdana,sans-serif;font-size:11px;color:rgb(17,85,204)">@AlbertoKlocker</a>
<div style="word-wrap:break-word">
<span style="border-collapse:separate">
<div style="word-wrap:break-word"><span
style="border-collapse:separate"><span
style="border-collapse:separate"><span
style="font-family:'Lucida Grande'">
<div style="margin:0cm 0cm
0.0001pt;font-size:12pt;font-family:'Times
New Roman',serif">
<span
style="border-collapse:separate;font-size:medium;font-family:Calibri"><span
style="font-family:'Times New
Roman',serif;font-size:16px"><span
style="border-collapse:separate;font-size:medium;font-family:Calibri">
<div style="margin:0cm 0cm
0.0001pt;font-size:12pt;font-family:'Times
New Roman',serif">
<span
style="font-size:8pt;font-family:Verdana,sans-serif;color:rgb(30,147,102)"
lang="EN-US">Please consider the
environment before printing this
email</span></div>
</span></span></span></div>
</span></span></span></div>
</span></div>
</span></div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@web-polygraph.org">Users@web-polygraph.org</a>
<a class="moz-txt-link-freetext" href="http://www.web-polygraph.org/mailman/listinfo/users">http://www.web-polygraph.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</body>
</html>