You can't charge the batteries that way. you have to build a constant current regulator using another LM317.
Ideally, you will mount an adjustable regulator with its milliammeter and adjust it to a tenth of the nominal current of the battery, allowing it to charge for 10 or 12 hours.
If you want I can give you the scheme.
I will tell you for the second time, you cannot charge a voltage battery unless it is an acid lead.
You can use the same power supply that you already had to regulate voltage, you would only have to connect another independent output than the lamellae of the output of the bridge of diodes.
It is assumed that if you have an adjustable source you already have an electrolytic capacitor at the exit of the bridge