tuning
Please review and implement the changes I suggested in my previous email. As soon as I can, I will schedule a call and assist DB.
If you really want to fix the issue immediately, re think the process you are going after. The system you have built will at max perform (with the changes) ~600-800 events per hour with ~9 actions per container. The more actions required by additional playbooks will slow the execution of containers exponentially.
Here are some performance configurations changes that can assist with locking issues and performance issues that have been explained to me below. This does not exclude the review of the playbook that “should” work while we are waiting to see if the performance changes will increase and improve the performance of the platform. Please let me know if we need another meeting after you have placed this in DEV, UAT and PROD. We should evaluate the playbook performance and go from there.
Apply the following changes to the Phantom node.
Modify the /opt/phantom/data/db/postgresql.conf with the changes below:
backup original .conf file
Verify Changes made for Med config (each sizing has a different config):
maxconnections = 300
sharedbuffers = 8GB
effectivecachesize = 24GB
maintenanceworkmem = 2GB
checkpointcompletiontarget = 0.9
walbuffers = 16MB
defaultstatisticstarget = 100
randompagecost = 1.1
effectiveioconcurrency = 200
workmem = 27962kB
checkpoint_segments = 64
systemctl restart postgresql-9.4
Update the /etc/nginx/uwsgi.ini file.
max-requests=100
workers=20
This will replace existing:
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
farm=ipc:2,3,4,5,6,7,8,9,10,11,12
Update the /etc/nginx/conf.d/default.conf file and add the following values to the server section.
clientbodybuffersize 10M;
clientheaderbuffersize 2k;
PGBouncer Tuning
Perform optional PGBouncer tuning on the Phantom node to handle more concurrent load. To perform PGBouncer tuning on the Phantom node, perform the following steps. In the /etc/pgbouncer/pgbouncer.ini file, change the connection poolmode from session to transaction.
Update the defaultpool_size to 300.
On the Splunk Phantom node, modify the limits.conf file according to the size and type of your configuration. To modify the limits.conf file, follow these steps.
Navigate to the following location on the Phantom node /etc/security/limits.conf and add the following code to the end of the limits.conf file.
- - memlock unlimited
- soft memlock unlimited
- hard memlock unlimited
- hard nofile 43690
- soft nofile 43690
- hard nproc 43690
- soft nproc 43690
root soft nproc unlimited
@postgres hard nofile 655360
@postgres soft nofile 655360
@postgres hard nproc 655360
@postgres soft nproc 655360
@pgbouncer hard nofile 655360
@pgbouncer soft nofile 655360
@pgbouncer hard nproc 655360
@pgbouncer soft nproc 655360
TCP Tuning
Perform TCP tuning to help optimize concurrency. To perform TCP tuning, run the following command:
sysctl -w net.core.somaxconn=4096
sysctl -w net.ipv4.tcpmaxsyn_backlog=4096
Kernel Tuning
Perform kernel tuning to achieve larger throughput and to help optimize concurrency. Update the kernel semaphore parameters and refresh the system configuration:
echo "kernel.sem=250 32000 32 5000" >> /etc/sysctl.conf
sysctl --system
Asset Pooling
Asset pooling allows All Phantom nodes to use multiple assets for playbooks. This will minimize the lock contention for playbooks to call your assets. You will need to create some special assets, add a asset.json file to use with playbook permission, and then some custom code in the global block to let you reutilize these assets.
Create the assets.json to the /opt/phantom/apps/ path by touch /opt/phantom/apps/assets.json
In the /opt/phantom/apps/assets.json, add the following for each application you are creating a pool for. (e.g. {"parser":5, “exchange”:6})
In the Phantom Applications menu create the assets for the applications that will support your app pool configuration.
The assets created must end with 000 to the number assets in your application pool. (e.g. parser000, parser001, parser002, parser003, parser004)
- Place the following global custom code in your playbooks that will use the application pools.
import random
def getrandomasset(app):
with open("/opt/phantom/apps/assetpools.json") as f:
assetpools = json.loads(f.read())
assetpoolsize = assetpools[app]
assetindex = random.choice(range(assetpoolsize))
assetname = app.replace(' ', '') + '' + '{0:03d}'.format(assetindex)
asset = [asset_name]
return asset
- When calling the phantom.act() change the asset=”parser” to asset=getrandomasset("parser")
Regards,