I've got a brand new combine harvester
1069 words
@vosillme

20 years man and boy (automation)

Curriculum Vitae

Profile

20 years as an IT Security Engineer helps me to:

  • focus on tasks and deliver them;
  • work well in teams;
  • make more informed decisions;
  • solve problems more rapidly and with less fuss; and
  • find and deliver solutions that require multiple teams to work together.

I am organized, positive, focused and work hard to ensure problems are untangled and projects are delivered.

SOAR experience

SOAR project summary

I am a member of the team assigned to deploy, manage and use Splunk Phantom to automate IT security related processes and orchestrate threat driven automations at my current customer, globally.

As a team we have successfully maintained a stable, redundant platform while deploying numerous and varied automations over the last 2 years.

We have 31 automations in production, have upgraded the redundant (using Warm-Standby) platform from versions 4.0 through 4.9 and have our first "Self-Service" user automations in production.

As a team we have promoted the platform internally via training days, Tableau dashboards for management reporting and hackathon type challenges.

I am designated single point of contact and responsible for internal Compliance and Regulatory adherence.

SOAR project examples

  • automation use case analysis (feasibility; usefulness) and design;
  • playbook delivery using GUI where possible, custom code (python) where necessary (example: bypass local RestAPI under load for direct to Postgres);
  • set-up and maintain Phantom Warm-Standby;
  • Geneos and Splunk application monitoring;
  • migrated Phantom to full clustering (7 nodes); and
  • SDLC approach using Development, UAT and Production environments; change management; HP ALM.

key skills

  • Network Security (Firewalls, Routers and Routing, Proxies, DNS)
  • RHEL and CentOS server management (filesystems, user management, repos, backups, patching)
  • IT Security Orchestration and Automation (Phantom, Cortex, use-case analysis, design and development)
  • python programming; html tagging; CSS and some javascript
  • Microsoft Azure (VM deployment; MS Graph; Microsoft365)
  • Google Cloud (VM Deployment)

career summary

  • 2010 - present: redacted
  • 2003 - 2010: redacted
  • 2002 - 2003: redacted
  • 1999 - 2002: redacted

education

BSc (hons) Computer Science degree from redacted

cv

phantom ps

tuning

Please review and implement the changes I suggested in my previous email. As soon as I can, I will schedule a call and assist DB.
If you really want to fix the issue immediately, re think the process you are going after. The system you have built will at max perform (with the changes) ~600-800 events per hour with ~9 actions per container. The more actions required by additional playbooks will slow the execution of containers exponentially.

Here are some performance configurations changes that can assist with locking issues and performance issues that have been explained to me below. This does not exclude the review of the playbook that “should” work while we are waiting to see if the performance changes will increase and improve the performance of the platform. Please let me know if we need another meeting after you have placed this in DEV, UAT and PROD. We should evaluate the playbook performance and go from there.

Apply the following changes to the Phantom node.

Modify the /opt/phantom/data/db/postgresql.conf with the changes below:

backup original .conf file

Verify Changes made for Med config (each sizing has a different config):
maxconnections = 300
shared
buffers = 8GB
effectivecachesize = 24GB
maintenanceworkmem = 2GB
checkpointcompletiontarget = 0.9
walbuffers = 16MB
default
statisticstarget = 100
random
pagecost = 1.1
effective
ioconcurrency = 200
work
mem = 27962kB

checkpoint_segments = 64

systemctl restart postgresql-9.4

Update the /etc/nginx/uwsgi.ini file.
max-requests=100
workers=20
This will replace existing:
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
mule=phantomui/ui/ipcmule.py
farm=ipc:2,3,4,5,6,7,8,9,10,11,12

Update the /etc/nginx/conf.d/default.conf file and add the following values to the server section.
clientbodybuffersize 10M;
client
headerbuffersize 2k;

PGBouncer Tuning
Perform optional PGBouncer tuning on the Phantom node to handle more concurrent load. To perform PGBouncer tuning on the Phantom node, perform the following steps. In the /etc/pgbouncer/pgbouncer.ini file, change the connection poolmode from session to transaction.
Update the default
pool_size to 300.

On the Splunk Phantom node, modify the limits.conf file according to the size and type of your configuration. To modify the limits.conf file, follow these steps.

Navigate to the following location on the Phantom node /etc/security/limits.conf and add the following code to the end of the limits.conf file.

  • - memlock unlimited
  • soft memlock unlimited
  • hard memlock unlimited
  • hard nofile 43690
  • soft nofile 43690
  • hard nproc 43690
  • soft nproc 43690 root soft nproc unlimited @postgres hard nofile 655360 @postgres soft nofile 655360 @postgres hard nproc 655360 @postgres soft nproc 655360 @pgbouncer hard nofile 655360 @pgbouncer soft nofile 655360 @pgbouncer hard nproc 655360 @pgbouncer soft nproc 655360

TCP Tuning
Perform TCP tuning to help optimize concurrency. To perform TCP tuning, run the following command:
sysctl -w net.core.somaxconn=4096
sysctl -w net.ipv4.tcpmaxsyn_backlog=4096

Kernel Tuning
Perform kernel tuning to achieve larger throughput and to help optimize concurrency. Update the kernel semaphore parameters and refresh the system configuration:
echo "kernel.sem=250 32000 32 5000" >> /etc/sysctl.conf
sysctl --system
Asset Pooling

Asset pooling allows All Phantom nodes to use multiple assets for playbooks. This will minimize the lock contention for playbooks to call your assets. You will need to create some special assets, add a asset.json file to use with playbook permission, and then some custom code in the global block to let you reutilize these assets.

Create the assets.json to the /opt/phantom/apps/ path by touch /opt/phantom/apps/assets.json
In the /opt/phantom/apps/assets.json, add the following for each application you are creating a pool for. (e.g. {"parser":5, “exchange”:6})
In the Phantom Applications menu create the assets for the applications that will support your app pool configuration.
The assets created must end with 000 to the number assets in your application pool. (e.g. parser000, parser001, parser002, parser003, parser004)

  1. Place the following global custom code in your playbooks that will use the application pools. import random def getrandomasset(app): with open("/opt/phantom/apps/assetpools.json") as f: assetpools = json.loads(f.read()) assetpoolsize = assetpools[app] assetindex = random.choice(range(assetpoolsize)) assetname = app.replace(' ', '') + '' + '{0:03d}'.format(assetindex) asset = [asset_name] return asset
  2. When calling the phantom.act() change the asset=”parser” to asset=getrandomasset("parser")

Regards,

listed

lest you dis me

links

https://docs.getpelican.com/en/stable/

for instance I coded this 1 or 2 years ago: https://github.com/cemonatk/simple-linkshortener flask is that easy... they both are python based + jinja templating.
But starting from django may seem like memorizing stuff.. By starting from Flask you could learn the HTTP/WebApp coding stugg easily. 👍🏻