In the previous paper [1], the author investigated the group structure of the homotopy set $[X,U(n)]$ with the pointwise multiplication, under the assumption that $X$ is a finite CW complex with its dimension $2n$ and $U(n)$ is the unitary group, and showed that $[X,U(n)]$ is an extension of $\Tilde{K}^{1}(X)$ by $N_{n}(X)$, where $N_{n}(X)$ is a group defined as the cokernel of a map $\Theta : \Tilde{K}^{0}(X)\to \mathrm{H}^{2n}(X;\mathbf{Z})$. In this paper, we offer another interpretation of $N_{n}(X)$ using Adams $e$-invariant and show that the extension $N_{n}(X) \to U_{n}(X) \to \Tilde{K}^{1}(X)$ is determined by some Toda brackets. Also we give some applications including the calculation of $[SO(4), U(3)]$.