Managing a single pool (continued)

(This post is part of a series on writing a process pool manager in Elixir.)

Continuing from the previous post, we’re working on having a single pool of processes that we can check out to do some work with. When we’re done, it should look like this:

The pool manager

Our pool manager is going to be a GenServer: it needs to be a process so it can maintain its state (e.g. tracking which worker processes are checked out). Let’s start with a super simplified version (lib/pool_toy/pool_man.ex):

defmodule PoolToy.PoolMan do
  use GenServer

  @name __MODULE__

  def start_link(size) when is_integer(size) and size > 0 do
    GenServer.start_link(__MODULE__, size, name: @name)
  end

  def init(size) do
    state = List.duplicate(:worker, size)
    {:ok, state}
  end
end

start_link/2 on line 6 takes a size indicating the number of workers we want to have in the pool. At this time, our pool manager is basically a trivial GenServer whose state consists of a list of identical atoms. These atoms represent workers we’ll implement later: right now, we’ll just use these atoms as a simple stand ins.

Let’s try it out and make sure it’s behaving as intended:

iex -S mix

iex(1)> PoolToy.PoolMan.start_link(3)
{:ok, #PID<0.112.0>}
iex(2)> :observer.start()
:ok

In the Observer, navigate to the pool manager process, and take a look at its state. Having trouble? Follow the steps in part 1.1. You can see that the process’ state is [worker, worker, worker] as expected (don’t forget, those are Erlang’s representation of atoms). Yay!

Our changes so far

And now, for our next trick, let’s have the pool supervisor start the pool manager upon startup.

Starting supervised children

This is what our pool supervisor currently has (lib/pool_toy/pool_sup.ex):

defmodule PoolToy.PoolSup do
  use Supervisor

  @name __MODULE__

  def start_link() do
    Supervisor.start_link(__MODULE__, [], name: @name)
  end

  def init([]) do
    children = []

    Supervisor.init(children, strategy: :one_for_all)
  end
end

We’ve got this convenient children value on line 11, let’s throw the pool manager value in there and see what happens: children = [PoolToy.PoolMan]. Within an IEx session (again, started with iex -S mix), try calling PoolToy.PoolSup.start_link() and you’ll get the following result:

** (EXIT from #PID<0.119.0>) shell process exited with reason: shutdown: failed to start child: PoolToy.PoolMan
    ** (EXIT) an exception was raised:
    ** (FunctionClauseError) no function clause matching in PoolToy.PoolMan.start_link/1
        (pool_toy) lib/pool_toy/pool_man.ex:6: PoolToy.PoolMan.start_link([])
        (stdlib) supervisor.erl:365: :supervisor.do_start_child/2
        (stdlib) supervisor.erl:348: :supervisor.start_children/3
        (stdlib) supervisor.erl:314: :supervisor.init_children/2
        (stdlib) gen_server.erl:365: :gen_server.init_it/2
        (stdlib) gen_server.erl:333: :gen_server.init_it/6
        (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

Darn. It looks like this programming stuff is going to be harder than just making random changes until we get something to work! Let’s take a closer look at the problem: the process couldn’t start PoolMan (line 1), because it was calling start_link([]) (line 4) but no function clause matched (line 3).

So how come PoolMan.start_link/1 is getting called with []? Let’s back up and see what Supervisor.init/2 does (docs). It refers us to more docs for additional information, where we’re told the first argument given to init/2 may be either:

• a map representing the child specification itself – as outlined in the “Child specification” section
• a tuple with a module as first element and the start argument as second – such as {Stack, [:hello]}. In this case, Stack.child_spec([:hello]) is called to retrieve the child specification
• a module – such as Stack. In this case, Stack.child_spec([]) is called to retrieve the child specification

What’s this about a child specification? It’s basically how a supervisor will (re)start and shutdown processes it supervises. Let’s leave it at that for now (but feel free to read more about child specs in the docs).

So based on what we’ve read, since we’re giving just a module within the children variable, at some point PoolMan.child_spec([]) gets called. This is pretty suspicious, because we haven’t defined a child_spec/1 function in the pool manager module. Let’s once again turn to the console to investigate what the heck is going on:

iex(1)> PoolToy.PoolMan.child_spec([])
%{id: PoolToy.PoolMan, start: {PoolToy.PoolMan, :start_link, [[]]}}
iex(2)> PoolToy.PoolMan.child_spec([foo: :bar])
%{id: PoolToy.PoolMan, start: {PoolToy.PoolMan, :start_link, [[foo: :bar]]}}

Let’s take another look at our pool manager module (lib/pool_toy/pool_man.ex):

defmodule PoolToy.PoolMan do
  use GenServer

  @name __MODULE__

  def start_link(size) when is_integer(size) and size > 0 do
    GenServer.start_link(__MODULE__, size, name: @name)
  end

  def init(size) do
    state = List.duplicate(:worker, size)
    {:ok, state}
  end
end

There’s definitely no child_spec/1 defined in there, so where’s it coming from? Well the only candidate is line 2, since use can generate code within our module. Sure enough, the GenServer docs tell us what happened:

use GenServer also defines a child_spec/1 function, allowing the defined module to be put under a supervision tree.

Now that that’s cleared up, let’s look at the resulting child spec again:

iex(2)> PoolToy.PoolMan.child_spec([foo: :bar])
%{id: PoolToy.PoolMan, start: {PoolToy.PoolMan, :start_link, [[foo: :bar]]}}

We’ve got an id attribute that the supervisor uses to differentiate its children: parents do the same thing by giving their kids different names, right? (Besides George Foreman and his sons, I mean.) The other key in there is used to define how the child should be started: as the docs indicate, it is a tuple containing the module and function to call, as well as the arguments to pass in. This is often referred to as an MFA (ie. Module, Function, Arguments). You’ll note the arguments are always wrapped within a single list. So in the example above where we want to pass a keyword list as an argument, it gets wrapped within another list in the MFA tuple.

Fixing the startup problem

Right, so after this scenic detour discussing child specs, let’s get back to our immediate problem:

1. If we give just a module name to Supervisor.init/2, it will call PoolMan.child_spec([]);
2. This will generate a child spec with a :start value of `{PoolToy.PoolMan, :start_link, [[]]}`;
3. The supervisor will attempt to call PoolMan‘s start_link function with [] as the argument;
4. The process crashes, because PoolMan only defines a start_link/1 function expecting an integer.

To fix this issue, we need the child spec’s start value to be something like `%{id: PoolToy.PoolMan, start: {PoolToy.PoolMan, :start_link, [3]}}` which would make the supervisor call PoolMan‘s start_link/1 function with size 3 and we’d get a pool with 3 workers.

How can this be solved? One option is to override the child_spec/1 function defined by use GenServer, to return the child spec we want, for example (lib/pool_toy/pool_man.ex):

def child_spec(_) do
    %{
      id: @name,
      start: {__MODULE__, :start_link, [3]}
    }
  end

That’ll work, but it’s not the best choice in our case: this isn’t flexible and is overkill for what we’re trying to achieve.

Another possibility is to customize the generated child_spec/1 function by altering the use GenServer statement on line 2 (lib/pool_toy/pool_man.ex):

use GenServer, start: {__MODULE__, :start_link, [3]}

Once again, not great: we want to be able to specify the pool size dynamically.

Referring back to the Supervisor.init/2 docs, we can see the other options we’ve got:

• a map representing the child specification itself – as outlined in the “Child specification” section
• a tuple with a module as first element and the start argument as second – such as {Stack, [:hello]}. In this case, Stack.child_spec([:hello]) is called to retrieve the child specification
• a module – such as Stack. In this case, Stack.child_spec([]) is called to retrieve the child specification

Per the first bullet point, we could also directly provide the child spec to Supervisor.init/2 in lib/pool_toy/pool_sup.ex:

defmodule PoolToy.PoolSup do
  use Supervisor

  @name __MODULE__

  def start_link(args) when is_list(args) do
    Supervisor.start_link(__MODULE__, args, name: @name)
  end

  def init(args) do
    pool_size = args |> Keyword.fetch!(:size)
    children = [
      %{
        id: PoolToy.PoolMan,
        start: {PoolToy.PoolMan, :start_link, [pool_size]}
      }
    ]

    Supervisor.init(children, strategy: :one_for_all)
  end
end

On line 6 we’ve modified start_link to take a keyword list with our options (as recommended by José) and forward it to init/1. In there, we extract the size value on line 11, and use that in our hand made child spec on line 15. In IEx, we can now call PoolToy.PoolSup.start_link(size: 5) and have our pool supervisor start up, and start its pool manager child. So this version also works, but it seems writing our own child spec is a lot of extra work when we were almost there using just the module name…

If look at the 2nd bullet point in the quoted docs above, you’ll find the simpler solution: use a tuple to specify the child spec. Just provide a tuple with the child module as the first element, and the startup args as the second element (lib/pool_toy/pool_sup.ex):

defmodule PoolToy.PoolSup do
  use Supervisor

  @name __MODULE__

  def start_link(args) when is_list(args) do
    Supervisor.start_link(__MODULE__, args, name: @name)
  end

  def init(args) do
    pool_size = args |> Keyword.fetch!(:size)
    children = [{PoolToy.PoolMan, pool_size}]

    Supervisor.init(children, strategy: :one_for_all)
  end
end

Back in our IEx shell, let’s make sure we didn’t break anything:

iex(1)> PoolToy.PoolSup.start_link(size: 5)
{:ok, #PID<0.121.0>}

Our changes so far

Another look at Observer

Let’s take a look in Obeserver with :observer.start(). Go to the “Processes” tab, find the Elixir.PoolToy.PoolMan process, and double-click it to open its process info window. If you now navigate to the “State” tab, you’ll see that we indeed have 5 worker atoms as the state: the size option was properly forwarded down to our pool manager from the pool supervisor! You can also see that there’s a parent pid on this screen: click it.

In the new window, you’ll be looking at the pool manager’s parent process: the window title indicates that its Elixir.PoolToy.PoolSup! So our pool supervisor did indeed start the pool manager as its child, as we intended. Finally, go to the “State” tab in this window for PoolSup and click on the “expand above term” link: you’ll see something like

{state,{local,'Elixir.PoolToy.PoolSup'},
       one_for_all,
       {['Elixir.PoolToy.PoolMan'],
        #{'Elixir.PoolToy.PoolMan' =>
              {child,<0.143.0>,'Elixir.PoolToy.PoolMan',
                     {'Elixir.PoolToy.PoolMan',start_link,[5]},
                     permanent,5000,worker,
                     ['Elixir.PoolToy.PoolMan']}}},
       undefined,3,5,[],0,'Elixir.PoolToy.PoolSup',
       [{size,5}]}

Once again, we’re poking our nose into something that we’re not really supposed to be aware of, but you can probably guess what a lot of information in there corresponds to: we’ve got a one_for_all restart strategy, a single childwith pid <0.143.0> (followed essentially by Erlang’s representation of the child spec used to start this particular child).

Continue on to the next post to add the worker supervisor and convert our app into an OTP application.

---

Would you like to see more Elixir content like this? Sign up to my mailing list so I can gauge how much interest there is in this type of content.