to know backpropagation?<\/p>\n
When you\u2019re attempting to know how trendy AI programs like massive language fashions (LLMs) are educated, backpropagation is without doubt one of the most essential ideas to know.<\/p>\n
However when you ask me how I felt once I encountered it, I used to be fully misplaced by wanting on the math equations. It felt like a psychological block for me.<\/p>\n
I noticed and wished to begin from scratch and construct my understanding one step at a time.<\/p>\n
That journey started with my earlier article, the place we constructed a neural community from scratch utilizing a easy dataset and understood the way it makes predictions.<\/p>\n
The weblog acquired an amazing response. Thanks for that!<\/p>\n
Now, let\u2019s proceed with the identical method. We’ll break down backpropagation step-by-step, maintaining it as easy and intuitive as earlier than.<\/p>\n
Earlier than we start, I simply wish to say one factor. We\u2019ll take this one step at a time. <\/p>\n
Subjects like backpropagation can really feel overwhelming at first, however as soon as we construct a powerful basis, all the things else turns into a lot simpler to know. <\/p>\n
So, let\u2019s get began.<\/p>\n
Welcome again!<\/p>\n
Let\u2019s proceed our studying journey via deep studying.<\/p>\n
We have already got a primary understanding of neural networks, which we explored utilizing a easy dataset within the earlier weblog.<\/p>\n
Now, let\u2019s first recall what we realized within the earlier weblog on neural networks.<\/p>\n
Fast Recap<\/h3>\n
We thought-about this straightforward dataset.<\/p>\n After plotting the info, it appeared like this:<\/p>\n We noticed {that a} single line was not sufficient to suit it. So, we determined to resolve it utilizing neural networks.<\/p>\n Subsequent, we obtained to know concerning the equation of a single neuron, and after that, we realized concerning the totally different layers in a neural community.<\/p>\n For simplicity, we thought-about one hidden layer with two hidden neurons.<\/p>\n Subsequent, we noticed how the 2 hidden neurons produced two totally different linear transformations, after which we wished to mix them within the output layer.<\/p>\n Nevertheless, we came upon that combining two strains produced one other line, not the curve that would match the info. <\/p>\n That is the place we realized the importance of activation capabilities, as they introduce non-linearity into the mannequin.<\/p>\n So, we handed the outputs from the hidden neurons via the activation perform (ReLU) after which mixed them within the output layer. <\/p>\n In different phrases, we took the linear mixture of the outputs from the activation perform within the output layer, and eventually, we obtained the curve.<\/p>\n Within the earlier weblog we constructed the neural community structure and noticed the way it makes predictions via ahead propagation.<\/p>\n Earlier than we learn the way backpropagation works, let\u2019s first take a look at the values produced at every layer throughout the ahead cross which we mentioned in earlier weblog. <\/p>\n We\u2019ll use these values all through the weblog to know how the community learns by updating its parameters.<\/p>\n Once we take a look at the ultimate curve produced by our neural community, we will see that it’s not a superb match. <\/p>\n For instance, when the hours studied (x) is 1, the precise examination rating is 55, however our neural community predicts it as 28, which is a large distinction.<\/p>\n Now, we have to make our neural community carry out higher, which suggests it ought to predict values which are a lot nearer to the precise examination scores. <\/p>\n To try this, the neural community must study. By studying, we imply determining which parameters must be elevated and which must be decreased to cut back the loss.<\/p>\n Now, how can we do that?<\/p>\n At this level, we don\u2019t know the way to do this.<\/p>\n Let\u2019s do one factor. Let\u2019s proceed with what we already know.<\/p>\n However what can we already know?<\/p>\n We have already got an thought about easy linear regression, how the loss is calculated, and the way the bowl curve appears to be like.<\/p>\n Possibly we will study one thing from it.<\/p>\n In easy linear regression, we have to discover the optimum values for \u03b20 (intercept) and \u03b21 (slope).<\/p>\n After all, we have already got formulation, however we additionally derived them ourselves.<\/p>\n What we did was plot a graph with three axes. One axis represented (\u03b20), the second represented (\u03b21), and the third represented the loss.<\/p>\n We plotted the loss values for various (\u03b20) and (\u03b21) values and noticed a bowl-shaped curve. <\/p>\n We then understood that the minimal loss happens on the backside of the curve, the place the slope of the loss floor turns into zero.<\/p>\n To search out that time, we used partial differentiation and ultimately solved the ensuing equations to acquire the formulation.<\/p>\n In easy linear regression, we will use totally different loss capabilities such because the Sum of Squared Errors (SSE), Imply Squared Error (MSE), or different appropriate loss capabilities relying on the issue.<\/p>\n Right here, we\u2019ll contemplate the Imply Squared Error (MSE) as our loss perform.<\/p>\n For easy linear regression, the loss perform is<\/p>\n [L(beta_0,beta_1)=frac{1}{n}sum_{i=1}^{n}left(y_i-hat{y}_iright)^2]<\/p>\n the place<\/p>\n [ Discover that the loss relies upon solely on two parameters, [beta_0] and [beta_1]<\/p>\n We have now to seek out the values of [beta_0] and [beta_1] that reduce this loss.<\/p>\n Now, let\u2019s take a look at our neural community.<\/p>\n Since our present drawback is a non-linear regression drawback, we will proceed utilizing the identical Imply Squared Error (MSE).<\/p>\n The loss perform can now be written as<\/p>\n [ Nevertheless, in contrast to easy linear regression, our prediction is not given by<\/p>\n [ As an alternative, it’s produced by the complete neural community.<\/p>\n For our neural community,<\/p>\n [ Consequently, the loss not depends upon simply two parameters. It now depends upon all seven parameters of the neural community, that are [w_1,w_2,w_3,w_4,b_1,b_2,b_3]<\/p>\n Similar to in easy linear regression, our purpose remains to be the identical: discover the values of those parameters that reduce the loss.<\/p>\n To attain that, we have to perceive how the loss modifications after we change every parameter individually whereas maintaining the remaining parameters mounted.<\/p>\n In different phrases, we have to compute partial derivatives akin to<\/p>\n [ These partial derivatives inform us how delicate the loss is to every parameter and assist us decide whether or not that parameter must be elevated or decreased to cut back the loss.<\/p>\n In easy linear regression, after we plot the loss values for various mixtures of the slope and intercept, we get a bowl-shaped curve in three-dimensional house.<\/p>\n For our neural community, nevertheless, we can’t visualize the loss floor in the identical method as a result of it now exists in eight-dimensional house. <\/p>\n Although we will\u2019t visualize it, our goal stays the identical which is to seek out the parameter values that reduce the loss.<\/p>\n Now, primarily based on what we already know from easy linear regression, we discovered a solution to proceed additional, which is to compute the partial derivatives of the loss with respect to every parameter.<\/p>\n The parameters are <\/p>\n [w_1,w_2,w_3,w_4,b_1,b_2,b_3]<\/p>\n However earlier than we proceed, there may be one essential idea that we have to perceive, and that’s the chain rule as a result of it’s the basis of all the things we\u2019re going to do subsequent.<\/p>\n The chain rule is used each time one amount depends upon one other amount, which in flip depends upon one other amount.<\/p>\n Let\u2019s perceive this with a easy instance.<\/p>\n Suppose<\/p>\n [ and<\/p>\n [ Now, we wish to discover<\/p>\n [ First, let\u2019s discover this by-product utilizing classical differentiation.<\/p>\n Discover that [z] is written when it comes to [y] not [x] Since we wish the by-product with respect to [x] we will first get rid of the intermediate variable by substituting [y=x^2] into the equation for [z]<\/p>\n Substituting,<\/p>\n [ Now the expression relies upon solely on [x] so we will differentiate it straight.<\/p>\n Utilizing the facility rule,<\/p>\n [ This technique might be straightforward for easy issues like this one as a result of we will simply substitute one expression into one other.<\/p>\n Nevertheless, think about a a lot greater expression with a number of intermediate variables. <\/p>\n Rewriting the complete equation earlier than differentiating would grow to be troublesome and there’s a larger likelihood for errors.<\/p>\n As an alternative of mixing all the things right into a single expression first, we now have a way more systematic method known as the chain rule. <\/p>\n As an alternative of eliminating the intermediate variables, with the chain rule we will work via them one step at a time.<\/p>\n Let\u2019s see how we will implement chain rule.<\/p>\n We already know that, [frac{dz}{dx}] tells us how a lot [z] modifications after we make a really small change in [x]<\/p>\n Right here [z] doesn’t rely straight on [x]<\/p>\n As an alternative, the connection appears to be like like this:<\/p>\n [ Which means each time [x] modifications, it first modifications [y] and that change in [y] then modifications [z]<\/p>\n Now as an alternative of attempting to distinguish all the things directly, the chain rule tells us to interrupt the issue into smaller items.<\/p>\n [ Now let\u2019s calculate every half individually.<\/p>\n Since<\/p>\n [ we get<\/p>\n [ Equally, since<\/p>\n [ we get<\/p>\n [ Multiplying these collectively,<\/p>\n [ Lastly, we all know that<\/p>\n [ so we substitute it again into the equation.<\/p>\n [ The essential factor we will observe right here is that we by no means differentiated the complete expression at a time.<\/p>\n As an alternative, we broke it into smaller derivatives, solved them one after the other, after which multiplied them collectively.<\/p>\n We\u2019ll use precisely the identical thought in our neural community.<\/p>\n The one distinction is that the chain is now a bit of longer.<\/p>\n Now that we perceive the chain rule, let\u2019s proceed to calculate the partial derivatives with respect to every parameter.<\/p>\n Till now, we used particular values for the weights and biases to know how the ahead cross works. Nevertheless, our goal is to study these values from the info.<\/p>\n So, as an alternative of utilizing mounted values, let\u2019s signify them utilizing parameters first.<\/p>\n The output of our neural community is given by<\/p>\n [hat{y}=w_3a_1+w_4a_2+b_3]<\/p>\n the place<\/p>\n [ and<\/p>\n [ Utilizing this output, we will calculate the Imply Squared Error (MSE), which is the loss perform of our neural community.<\/p>\n The final MSE equation is<\/p>\n [ Now, let\u2019s substitute the prediction equation into the loss perform.<\/p>\n [ Since<\/p>\n [ and<\/p>\n [ the entire loss perform turns into<\/p>\n [ Now, let\u2019s begin discovering the partial by-product with respect to any one of many parameters, let\u2019s start with<\/p>\n [w_1]<\/p>\n So, we now have to calculate<\/p>\n [frac{partial}{partial w_1}left[frac{1}{n}sum_{i=1}^{n}left(y_i-left(w_3,mathrm{ReLU}(w_1x_i+b_1)+w_4,mathrm{ReLU}(w_2x_i+b_2)+b_3right)right)^2right]]<\/p>\n This equation appears troublesome to resolve. How can we discover the partial by-product with respect to<\/p>\n [w_1]<\/p>\n from such a big equation?<\/p>\n Let\u2019s proceed utilizing the identical concepts from differentiation that we already know.<\/p>\n As an alternative of differentiating all the things directly, we\u2019ll simplify the issue step-by-step.<\/p>\n We wish to compute<\/p>\n [ Substitute the loss perform into the by-product.<\/p>\n [ Discover that we don’t substitute the complete expression for <\/p>\n [hat{y}_i]<\/p>\n but. We’ll try this solely when it turns into needed.<\/p>\n As [frac{1}{n}] is a continuing, we all know that it may be moved outdoors the by-product.<\/p>\n [ The summation can also be linear, so the by-product can cross via it.<\/p>\n What can we imply by that?<\/p>\n It means in summation we add many phrases collectively and we will differentiate every time period individually after which add the derivatives.<\/p>\n [frac{partial L}{partial w_1}=frac{1}{n}sum_{i=1}^{n}frac{partial}{partial w_1}left((y_i-hat{y}_i)^2right)]<\/p>\n Now Differentiate the Sq.<\/p>\n Let<\/p>\n [ Then<\/p>\n [ Utilizing the facility rule,<\/p>\n [ Substitute this into the earlier equation.<\/p>\n [ Change [A] with [y_i-hat{y}_i].<\/p>\n [ Differentiate the Expression Inside<\/p>\n We all know that the true goal (precise commentary) [y_i] is a continuing,<\/p>\n [ Subsequently,<\/p>\n [ Substitute this again.<\/p>\n [ Now Differentiating the Prediction [frac{partialhat{y}_i}{partial w_1}]<\/p>\n We all know that from the output layer in our neural community<\/p>\n [ Substitute the equations of hidden neuron activation capabilities.<\/p>\n [ Differentiate with respect to [w_1]<\/p>\n [ Differentiating every time period individually.<\/p>\n As [w_3] is fixed,<\/p>\n [ The second time period incorporates solely [w_2] so<\/p>\n [ Additionally,<\/p>\n [ Therefore,<\/p>\n [ Differentiate the ReLU Expression<\/p>\n Let<\/p>\n [ Then<\/p>\n [ Now Differentiating<\/p>\n [ with respect to [w_1]<\/p>\n [ Now differentiate the activation.<\/p>\n [ Right here we use the chain rule,<\/p>\n [ Substituting [frac{du}{dw_1}=x_i]<\/p>\n [ Changing [u]<\/p>\n [ This ReLU derivation would possibly get complicated, let\u2019s decelerate and see what we truly did right here.<\/p>\n We all know that the ReLU activation doesn\u2019t rely on w1w_1 straight. <\/p>\n It depends upon the worth of w1xi+b1w_1x_i+b_1\u200b. <\/p>\n On the similar time, the expression w1xi+b1w_1x_i+b_1\u200b depends upon w1w_1\u200b. <\/p>\n So when w1w_1 modifications, it first modifications w1xi+b1w_1x_i+b_1, which in flip modifications the output of the ReLU. <\/p>\n That is precisely the type of scenario the place we use the chain rule.<\/p>\n So to seek out how the ReLU modifications with respect to w1w_1\u200b, we first discover how w1xi+b1w_1x_i+b_1 modifications with respect to w1w_1, after which how the ReLU modifications with respect to w1xi+b1w_1x_i+b_1\u200b. <\/p>\n Lastly, we mix these two outcomes utilizing the chain rule.<\/p>\n Now Substitute Again<\/p>\n Earlier, we discovered<\/p>\n [ We additionally calculated<\/p>\n [ Substitute this into the earlier equation.<\/p>\n [ We have now derived<\/p>\n [ This tells us precisely how the loss modifications when the burden<\/p>\n [w_1]<\/p>\n modifications.<\/p>\n At first, this equation might look obscure, however it’s truly fairly easy.<\/p>\n Because it tells us how the overall loss modifications after we make a really small change to the burden w1\u200b, this worth is known as the gradient, and it’s precisely what gradient descent makes use of to replace the burden.<\/p>\n To calculate this gradient, the equation considers each coaching instance within the dataset.<\/p>\n For every coaching instance:<\/p>\n [(y_i-hat{y}_i)] tells us how far the prediction is from the precise worth.<\/p>\n [w_3] tells us how a lot the primary hidden neuron contributes to the ultimate prediction.<\/p>\n [mathrm{ReLU}'(w_1x_i+b_1)] tells us whether or not the change in [w_1] can cross via the ReLU activation.<\/p>\n [x_i] tells us how a lot a small change in [w_1] impacts the neuron\u2019s enter.<\/p>\n Every coaching instance contributes its personal gradient primarily based on these portions. <\/p>\n We add all of those particular person contributions collectively, and since we’re utilizing the Imply Squared Error (MSE) loss perform, dividing by nn offers the typical gradient throughout the complete dataset.<\/p>\n This common gradient tells us how w1w_1 must be adjusted to cut back the general loss, fairly than simply the error for a single coaching instance.<\/p>\n When you keep in mind our dialogue on Easy Linear Regression, we calculated the partial derivatives with respect to solely two parameters.<\/p>\n On this weblog, we now have efficiently derived<\/p>\n [frac{partial L}{partial w_1}]<\/p>\n Though the derivation was lengthy, We used the identical concepts from calculus that we already knew in each step. <\/p>\n We merely utilized differentiation step-by-step and used the chain rule wherever it was required.<\/p>\n Now, our neural community nonetheless has six extra parameters, and every of them has its personal partial by-product.<\/p>\n So, what do you suppose?<\/p>\n Do we have to repeat this whole course of for each weight and bias?<\/p>\n Luckily, no.<\/p>\n As neural networks grow to be bigger, manually deriving each gradient would shortly grow to be exhausting and inefficient.<\/p>\n There must be a greater method.<\/p>\n The excellent news is that we don\u2019t want any new arithmetic. <\/p>\n We merely want a greater solution to arrange these calculations. <\/p>\n Every thing remains to be constructed on the identical chain rule we\u2019ve been utilizing all through this text.<\/p>\n Within the subsequent half, we\u2019ll see how the chain rule might be utilized effectively throughout the complete neural community, main us to one of the crucial essential algorithms in deep studying: backpropagation.<\/p>\n I hope you realized one thing from this text. When you\u2019re nonetheless confused about neural networks or wish to revisit the fundamentals, you’ll be able to all the time learn my earlier article right here.<\/p>\n When you discovered this useful, be at liberty to share it with individuals who might have it.<\/p>\n![\"\"](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/07\/backpropagation-blog.png\")
![\"\"](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/06\/exam_scores_scatter-1024x826.png\")
![\"\"](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/06\/final_neural_network_output-1024x609.png\")
![\"\"](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/06\/NEURAL-NETWORKS-11.png\")
![\"\"](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/07\/backpropagation-blog-1-1024x358.png\")
Why Does the Community Must Be taught?<\/h3>\n
Studying from a Acquainted Instance<\/h3>\n
![\"\"](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/07\/loss_surface-1002x1024.png\")
\nhat{y}_i=beta_0+beta_1x_i.
\n]<\/p>\n
\nL(w_1,w_2,w_3,w_4,b_1,b_2,b_3)
\n=
\nfrac{1}{n}
\nsum_{i=1}^{n}
\nleft(y_i-hat{y}_iright)^2.
\n]<\/p>\n
\nhat{y}=beta_0+beta_1x.
\n]<\/p>\n
\nhat{y}_i
\n=
\nw_3,mathrm{ReLU}(w_1x_i+b_1)
\n+
\nw_4,mathrm{ReLU}(w_2x_i+b_2)
\n+
\nb_3.
\n]<\/p>\n
\nfrac{partial L}{partial w_1},
\nquad
\nfrac{partial L}{partial w_2},
\nquad
\nfrac{partial L}{partial w_3},
\nquad
\nldots,
\nquad
\nfrac{partial L}{partial b_3}.
\n]<\/p>\nSetting the Purpose<\/h3>\n
Time to Perceive Chain Rule<\/h3>\n
\ny=x^2
\n]<\/p>\n
\nz=y^3
\n]<\/p>\n
\nfrac{dz}{dx}
\n]<\/p>\n
\nz=(x^2)^3=x^6
\n]<\/p>\n
\nfrac{dz}{dx}
\n=
\nfrac{d}{dx}(x^6)
\n=
\n6x^5
\n]<\/p>\n
\nx rightarrow y rightarrow z.
\n]<\/p>\n
\nfrac{dz}{dx}=frac{dz}{dy}timesfrac{dy}{dx}
\n]<\/p>\n
\nz=y^3,
\n]<\/p>\n
\nfrac{dz}{dy}=3y^2.
\n]<\/p>\n
\ny=x^2,
\n]<\/p>\n
\nfrac{dy}{dx}=2x.
\n]<\/p>\n
\nfrac{dz}{dx}=3y^2times2x.
\n]<\/p>\n
\ny=x^2,
\n]<\/p>\n
\nfrac{dz}{dx}=3(x^2)^2times2x=6x^5.
\n]<\/p>\nFixing step-by-step utilizing Classical Differentiation Technique<\/h3>\n
\na_1=mathrm{ReLU}(z_1)
\n]
\n[
\nz_1=w_1x+b_1
\n]<\/p>\n
\na_2=mathrm{ReLU}(z_2)
\n]
\n[
\nz_2=w_2x+b_2
\n]<\/p>\n
\nL(w_1,w_2,w_3,w_4,b_1,b_2,b_3)=frac{1}{n}sum_{i=1}^{n}(y_i-hat{y}_i)^2
\n]<\/p>\n
\nL(w_1,w_2,w_3,w_4,b_1,b_2,b_3)=frac{1}{n}sum_{i=1}^{n}left(y_i-left(w_3a_{1i}+w_4a_{2i}+b_3right)right)^2
\n]<\/p>\n
\na_{1i}=mathrm{ReLU}(w_1x_i+b_1)
\n]<\/p>\n
\na_{2i}=mathrm{ReLU}(w_2x_i+b_2)
\n]<\/p>\n
\nL(w_1,w_2,w_3,w_4,b_1,b_2,b_3)=frac{1}{n}sum_{i=1}^{n}left(y_i-left(w_3mathrm{ReLU}(w_1x_i+b_1)+w_4mathrm{ReLU}(w_2x_i+b_2)+b_3right)right)^2
\n]<\/p>\n
\nfrac{partial L}{partial w_1}
\n]<\/p>\n
\nfrac{partial L}{partial w_1}
\n=
\nfrac{partial}{partial w_1}
\nleft(
\nfrac{1}{n}
\nsum_{i=1}^{n}
\n(y_i-hat{y}_i)^2
\nright)
\n]<\/p>\n
\nfrac{partial L}{partial w_1}
\n=
\nfrac{1}{n}
\nfrac{partial}{partial w_1}
\nleft(
\nsum_{i=1}^{n}
\n(y_i-hat{y}_i)^2
\nright)
\n]<\/p>\n
\nA=y_i-hat{y}_i
\n]<\/p>\n
\nfrac{partial L}{partial w_1}
\n=
\nfrac{1}{n}
\nsum_{i=1}^{n}
\nfrac{partial}{partial w_1}(A^2)
\n]<\/p>\n
\nfrac{partial}{partial w_1}(A^2)
\n=
\n2A
\nfrac{partial A}{partial w_1}
\n]<\/p>\n
\nfrac{partial L}{partial w_1}
\n=
\nfrac{2}{n}
\nsum_{i=1}^{n}
\nA
\nfrac{partial A}{partial w_1}
\n]<\/p>\n
\nfrac{partial L}{partial w_1}
\n=
\nfrac{2}{n}
\nsum_{i=1}^{n}
\n(y_i-hat{y}_i)
\nfrac{partial}{partial w_1}
\n(y_i-hat{y}_i)
\n]<\/p>\n
\nfrac{partial y_i}{partial w_1}=0
\n]<\/p>\n
\nfrac{partial}{partial w_1}
\n(y_i-hat{y}_i)
\n=
\n-frac{partialhat{y}_i}{partial w_1}
\n]<\/p>\n
\nfrac{partial L}{partial w_1}
\n=
\n-frac{2}{n}
\nsum_{i=1}^{n}
\n(y_i-hat{y}_i)
\nfrac{partialhat{y}_i}{partial w_1}
\n]<\/p>\n
\nhat{y}_i
\n=
\nw_3a_{1i}
\n+
\nw_4a_{2i}
\n+
\nb_3
\n]<\/p>\n
\nhat{y}_i
\n=
\nw_3mathrm{ReLU}(w_1x_i+b_1)
\n+
\nw_4mathrm{ReLU}(w_2x_i+b_2)
\n+
\nb_3
\n]<\/p>\n
\nfrac{partialhat{y}_i}{partial w_1}
\n=
\nfrac{partial}{partial w_1}
\nleft(
\nw_3mathrm{ReLU}(w_1x_i+b_1)
\n+
\nw_4mathrm{ReLU}(w_2x_i+b_2)
\n+
\nb_3
\nright)
\n]<\/p>\n
\nfrac{partial}{partial w_1}
\nleft(
\nw_3mathrm{ReLU}(w_1x_i+b_1)
\nright)
\n=
\nw_3
\nfrac{partial}{partial w_1}
\nmathrm{ReLU}(w_1x_i+b_1)
\n]<\/p>\n
\nfrac{partial}{partial w_1}
\nleft(
\nw_4mathrm{ReLU}(w_2x_i+b_2)
\nright)
\n=0
\n]<\/p>\n
\nfrac{partial b_3}{partial w_1}=0
\n]<\/p>\n
\nfrac{partialhat{y}_i}{partial w_1}
\n=
\nw_3
\nfrac{partial}{partial w_1}
\nmathrm{ReLU}(w_1x_i+b_1)
\n]<\/p>\n
\nu=w_1x_i+b_1
\n]<\/p>\n
\nmathrm{ReLU}(w_1x_i+b_1)
\n=
\nmathrm{ReLU}(u)
\n]<\/p>\n
\nu=w_1x_i+b_1
\n]<\/p>\n
\nfrac{du}{dw_1}=x_i
\n]<\/p>\n
\nfrac{d,mathrm{ReLU}(u)}{du}
\n=
\nmathrm{ReLU}'(u)
\n]<\/p>\n
\nfrac{partial}{partial w_1}
\nmathrm{ReLU}(w_1x_i+b_1)
\n=
\nmathrm{ReLU}'(u)
\nfrac{du}{dw_1}
\n]<\/p>\n
\nfrac{partial}{partial w_1}
\nmathrm{ReLU}(w_1x_i+b_1)
\n=
\nmathrm{ReLU}'(u)x_i
\n]<\/p>\n
\nfrac{partial}{partial w_1}
\nmathrm{ReLU}(w_1x_i+b_1)
\n=
\nmathrm{ReLU}'(w_1x_i+b_1)x_i
\n]<\/p>\n
\nfrac{partial L}{partial w_1}
\n=
\n-frac{2}{n}
\nsum_{i=1}^{n}
\n(y_i-hat{y}_i)
\nfrac{partialhat{y}_i}{partial w_1}
\n]<\/p>\n
\nfrac{partialhat{y}_i}{partial w_1}
\n=
\nw_3
\nmathrm{ReLU}'(w_1x_i+b_1)
\nx_i
\n]<\/p>\n
\nfrac{partial L}{partial w_1}
\n=
\n-frac{2}{n}
\nsum_{i=1}^{n}
\n(y_i-hat{y}_i)
\nw_3
\nmathrm{ReLU}'(w_1x_i+b_1)
\nx_i
\n]<\/p>\nClosing outcome<\/h3>\n
\nfrac{partial L}{partial w_1}
\n=
\n-frac{2}{n}
\nsum_{i=1}^{n}
\n(y_i-hat{y}_i)
\nw_3
\nmathrm{ReLU}'(w_1x_i+b_1)
\nx_i
\n]<\/p>\nConclusion<\/h3>\n