Terms

Vocab and terminology to have a good grasp of.

Moore's Law

The number of transistors on an integrated circuit doubles approximately every two years. This exponential rate of growth is what has been driving force behind the massive amounts of computing and technology development in recent years, with now inexpensive compute deployed at scale.

Detail

A integrated circuit (IC) or microchip or chip, is made of semiconductor material (silicon) with transistors in the silicon wired together.

Amdahl's Law

Amdahl's law states that the overall speedup of a task is limited by the portion of the task that cannot be parallelized. Even if you had infinite parallel processers, you are limited to the sequential part of the computation.

The law allows you to compute the speedup of a task given processors or workers:

$$\text{speedup} = \frac{\text{walltime of serial execution}}{\text{walltime of parallel execution}} = \frac{1}{(1-p) + \frac{p}{n}}$$

Where $p$ is the fraction of the code that can be parallelized and $n$ is number of workers. You can think of it as time with 1 worker divided by time in non-parallelizable part $1-p$ added to the parallelized time $\frac{p}{n}$. For a certain percentage of code that can be sped up, you are fundamentally capped to a theoretical speedup limit. As $n$ tends to infinity, you approach the asymptotic $\frac{1}{1-p}$ limit. If only 50% of the code is parallelizable, you can at most have a 2x speedup.

Monte-Carlo Simulation

Monte-Carlo methods use repeated random sampling to estimate the probability of a range of events occurring. For example, if you wanted to estimate the probability of getting 12 from a two dice roll, you simulate the dice rolling 10,000 times to get an accurate estimate. With predictive models you will have independent variables you generate random values for and dependent variables to be predicted based off those independent variables. You run simulations each time modifying the underlying params.

Likert Scale

Qualitative number scales e.g. "Rate X from 1 (bad) to 5 (good)".

Mainframe

Mainframes are large powerful computers built in the 1960s-1980s to handle large volumes of transactions and computations extremely reliably. They are built to run 24/7 without crashing.

I overheard an Amazon exec talk about how many companies are still using legacy mainframes because they work so well. He was using this as evidence on the point that many enterprises choose to stay with what works over adopting new flashy technologies, especially those touting AI transformations.

Amortized

Something being amortized meaning that the cost has been gradually written off over a period of time.

Example: "The price should be quite low because all the R&D has long since been amortized"

HPC Schedulers

In HPC a batch scheduler is used to divvy up resources and determine who gets what compute resources for how long. Generally you submit a request to the scheduler with info on the time, compute, and memory resources you need for your job.