From Biological to Artificial Neurons
0% read · 6 min left
From Biological to Artificial Neurons
Jr Codex Deep Learning Notes
Level: Beginner Prerequisites: Chapter 1: What Is Deep Learning & Why It Works Time to complete: ~20 minutes
Table of Contents
- The Biological Inspiration
- The Artificial Neuron
- Weights, Bias, and What They Mean
- Why a Single Neuron Isn't Enough
- From One Neuron to a Network
- A Neuron in Code
- Summary & Next Steps
1. The Biological Inspiration
Artificial neural networks are loosely inspired by how biological neurons work, though the analogy is much looser than it's often presented — it's worth understanding both the inspiration and its limits.
A Biological Neuron (Simplified)
─────────────────────────────────────────
Dendrites: receive electrical/chemical signals from
OTHER neurons
Cell body: accumulates incoming signals
Threshold: if accumulated signal exceeds a
threshold, the neuron "FIRES"
Axon: the fired signal travels onward
to the NEXT neurons
─────────────────────────────────────────
The key idea borrowed from biology: a neuron combines multiple inputs, and produces an output only if the combined signal is strong enough. Everything else — the "artificial" part — is a mathematical simplification designed for computation and optimization, not a faithful simulation of real neuroscience.
2. The Artificial Neuron
An artificial neuron (also called a unit or perceptron, historically) does exactly one thing: it computes a weighted sum of its inputs, adds a bias, and passes the result through an activation function.
x1 ──(w1)──┐
x2 ──(w2)──┼──► Σ (weighted sum + bias) ──► f(·) ──► output
x3 ──(w3)──┘ z = w1·x1 + w2·x2 + w3·x3 + b activation
function
import numpy as np
def artificial_neuron(inputs, weights, bias, activation_fn):
z = np.dot(inputs, weights) + bias # the weighted sum + bias
return activation_fn(z) # the activation function
def step_function(z):
"""The ORIGINAL 1950s activation — outputs 0 or 1"""
return 1 if z > 0 else 0
inputs = np.array([0.5, 0.3, 0.9])
weights = np.array([0.4, -0.2, 0.6])
bias = -0.1
output = artificial_neuron(inputs, weights, bias, step_function)
print(f"Neuron output: {output}")3. Weights, Bias, and What They Mean
What Each Piece Represents
─────────────────────────────────────────
Weights (w1, w2, w3, ...): how MUCH each input matters —
a large positive weight means
that input strongly pushes
the neuron toward firing; a
large negative weight strongly
pushes AGAINST it
Bias (b): shifts the threshold for
firing — independent of any
input, it makes the neuron
easier or harder to activate
overall
Activation function f(·): introduces NON-LINEARITY
(Section 4) — without it,
stacking neurons would be
mathematically pointless
─────────────────────────────────────────
Weights and biases are exactly what "training a neural network" means finding. Every value that gets adjusted during training (Chapter 4's backpropagation) is a weight or a bias — nothing else in the network changes.
4. Why a Single Neuron Isn't Enough
A single neuron with a linear weighted sum, even with a non-linear activation, can only separate data with a single straight line (or hyperplane in higher dimensions) — this is mathematically identical to logistic regression from the ML Notes.
The Classic Limitation: XOR
─────────────────────────────────────────
Input (A, B) Output (A XOR B)
(0, 0) 0
(0, 1) 1
(1, 0) 1
(1, 1) 0
Plot these four points — NO single straight line can
separate the 0s from the 1s. A single neuron (or logistic
regression) CANNOT learn XOR, no matter how it's trained.
─────────────────────────────────────────
This limitation, identified in 1969, caused the first major "AI winter" for neural networks (covered in Chapter 3) — it took over a decade before the field recovered from it, once multiple layers of neurons were shown to solve exactly this problem.
5. From One Neuron to a Network
Stacking neurons into layers, where each layer's outputs feed into the next layer's inputs, is what actually enables learning complex, non-linear patterns like XOR.
A Minimal Network Solving XOR
─────────────────────────────────────────
Input Layer Hidden Layer Output Layer
(2 neurons) (2 neurons) (1 neuron)
x1 ──┬────► h1 ────┐
│ ├──► output
x2 ──┴────► h2 ────┘
Each hidden neuron learns a DIFFERENT linear boundary;
the output neuron COMBINES them non-linearly, carving out
a region no single line could represent.
─────────────────────────────────────────
This structure — an input layer, one or more hidden layers, and an output layer — is called a Multi-Layer Perceptron (MLP), the most basic deep learning architecture, covered in full in Module 2.
6. A Neuron in Code
import numpy as np
class Neuron:
def __init__(self, num_inputs):
# Small random initial weights — NOT zero (Module 3 explains why)
self.weights = np.random.randn(num_inputs) * 0.1
self.bias = 0.0
def forward(self, inputs):
z = np.dot(inputs, self.weights) + self.bias
return self.sigmoid(z) # a smoother activation than the step function
def sigmoid(self, z):
return 1 / (1 + np.exp(-z))
neuron = Neuron(num_inputs=3)
sample_input = np.array([0.5, 0.3, 0.9])
print(f"Neuron output: {neuron.forward(sample_input):.4f}")This single Neuron class, replicated and connected in layers, is the entire building block for everything through Module 8 — CNNs, RNNs, and Transformers are all, at their core, specific ways of arranging and connecting neurons like this one.
7. Summary & Next Steps
Key Takeaways
- An artificial neuron computes a weighted sum of its inputs plus a bias, then applies an activation function — a loose, simplified analogy to biological neurons.
- Weights determine how much each input matters; the bias shifts the firing threshold; the activation function introduces non-linearity.
- A single neuron (like logistic regression) can only separate data linearly — it cannot solve non-linearly separable problems like XOR.
- Stacking neurons into layers (an MLP) enables learning non-linear patterns, which is the foundation of every deep learning architecture that follows.
Concept Check
- What do the "weights" of a neuron actually represent, and what does training a network mean in terms of these weights?
- Why can't a single neuron solve the XOR problem, and what changes when you add a hidden layer?
- Which classical ML algorithm is mathematically equivalent to a single neuron with a sigmoid activation?
Next Chapter
→ Chapter 3: A Brief History of Deep Learning
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index