opnsense on FW4B Supplicant

wsciaroni commented

2021-04-06 04:04:23 +05:30

(Migrated from github.com)

Hello,

I've been troubleshooting this for several days now and am finally coming with an issue. I'd be happy to provide more info.

I have previously used bridge mode with pfsense at two other installs without any issues! Thank you for all the work you put into this project.

What I'm working with

However, I have migrated to opnsense when I moved and I'm having trouble. Here's what I'm using:

Software

OPNsense 21.1.4-amd64
FreeBSD 12.1-RELEASE-p15-HBSD
OpenSSL 1.1.1k 25 Mar 2021

Hardware

Protectli FW4B (With 4g failover Nic shows as a USB device ue0)

BGW210-700
XSGPON ONT (one of the two sites I've done this previously had the same ONT)

What I'm seeing

I've got the certs and I'm using the mac from the certs.
I've done packet captures and made sure that I've entered the correct EAP identity in the script
I've tried matching permissions of the certs with no luck there either. I've tried 600 666 and 700 as recommended.

When I take a capture of the [RG] -- [ONT] traffic I am able to capture the entire EAP handshake. I noticed that none of the EAP traffic are tagged on vlan0.

However, when I attempt [opnsense] -- [ont] and capture that traffic, my EAP outbound start packet is being tagged on vlan0. This is causing a loop in which both devices are ready to start authentication, but I'm on vlan0 and they're waiting around looking for untagged traffic.

At least that's what I suspect is happening as it's the only difference I can really tell. So, essentially I'm never seeing the EAP traffic progress past a EAP Request Identity and EAP Start.

Additional remarks:

Packet Capture Method

I wanted to add that I'm capturing the packets using a switch set up to mirror 3 ports. In this way I have the RG, ONT, and capture interface on laptop tied to the switch when capturing [RG] -- [ONT].
I have ONT, Opnsense, and capture interface on laptop tied to the switch when capturing [Opnsense]--[ONT].

What I'm capturing

The actual EAPOL start packet that I captured can be seen below. Clearly somehow it is getting tagged...

When it's [RG]--[ONT] I see the following in the start frame (Notice no tagging)

I am able to observe the entire EAP handshake for [RG]--[ONT]

Absolutely forgot to mention that I'm using the opnatt.sh from the supplicant_OPNsense_testing branch.Absolutely forgot to mention that I'm using the opnatt.sh from the supplicant_OPNsense_testing branch.

Conclusion

Thoughts?

Hello, I've been troubleshooting this for several days now and am finally coming with an issue. I'd be happy to provide more info. I have previously used bridge mode with pfsense at two other installs without any issues! Thank you for all the work you put into this project. ## What I'm working with However, I have migrated to opnsense when I moved and I'm having trouble. Here's what I'm using: ### Software ```txt OPNsense 21.1.4-amd64 FreeBSD 12.1-RELEASE-p15-HBSD OpenSSL 1.1.1k 25 Mar 2021 ``` ### Hardware Protectli FW4B (With 4g failover Nic shows as a USB device `ue0`) BGW210-700 XSGPON ONT (one of the two sites I've done this previously had the same ONT) ## What I'm seeing - I've got the certs and I'm using the mac from the certs. - I've done packet captures and made sure that I've entered the correct EAP identity in the script - I've tried matching permissions of the certs with no luck there either. I've tried `600` `666` and `700` as recommended. When I take a capture of the [RG] -- [ONT] traffic I am able to capture the entire EAP handshake. I noticed that none of the EAP traffic are tagged on `vlan0`. However, when I attempt [opnsense] -- [ont] and capture that traffic, my EAP outbound `start` packet is being tagged on `vlan0`. This is causing a loop in which both devices are ready to start authentication, but I'm on vlan0 and they're waiting around looking for untagged traffic. At least that's what I suspect is happening as it's the only difference I can really tell. So, essentially I'm never seeing the EAP traffic progress past a `EAP Request Identity` and `EAP Start`. ![image](https://user-images.githubusercontent.com/26583386/113634812-0d916e80-9635-11eb-8544-746223d172aa.png) ## Additional remarks: ## Packet Capture Method I wanted to add that I'm capturing the packets using a switch set up to mirror 3 ports. In this way I have the RG, ONT, and capture interface on laptop tied to the switch when capturing [RG] -- [ONT]. I have ONT, Opnsense, and capture interface on laptop tied to the switch when capturing [Opnsense]--[ONT]. ## What I'm capturing The actual EAPOL start packet that I captured can be seen below. Clearly somehow it is getting tagged... ![image](https://user-images.githubusercontent.com/26583386/113635501-6281b480-9636-11eb-834c-df42e8bea2d7.png) When it's [RG]--[ONT] I see the following in the start frame (Notice no tagging) ![image](https://user-images.githubusercontent.com/26583386/113635659-a4aaf600-9636-11eb-819b-948dc3c9604a.png) I am able to observe the entire EAP handshake for [RG]--[ONT] ![image](https://user-images.githubusercontent.com/26583386/113635767-d754ee80-9636-11eb-9f5d-81bd410cd9d6.png) Absolutely forgot to mention that I'm using the `opnatt.sh` from the `supplicant_OPNsense_testing` branch.Absolutely forgot to mention that I'm using the `opnatt.sh` from the `supplicant_OPNsense_testing` branch. ## Conclusion Thoughts?

cspencer49519 commented

2021-05-13 10:12:31 +05:30

(Migrated from github.com)

Have you tried manually executing opnatt.sh after the device has fully booted? I haven't logged it, but I'm on one version previous to you:

OPNsense 21.1.3_3-amd64FreeBSD 12.1-RELEASE-p14-HBSDOpenSSL 1.1.1j 16 Feb 2021

Since I updated to this version, anytime I reboot my router I need to execute the script manually after reboot to get everything up and running.

Have you tried manually executing opnatt.sh after the device has fully booted? I haven't logged it, but I'm on one version previous to you: OPNsense 21.1.3_3-amd64FreeBSD 12.1-RELEASE-p14-HBSDOpenSSL 1.1.1j 16 Feb 2021 Since I updated to this version, anytime I reboot my router I need to execute the script manually after reboot to get everything up and running.

wsciaroni commented

2021-07-17 07:45:28 +05:30

(Migrated from github.com)

I have manually run the script as well as run each line of the script manually. Once I get the script running I see that the handshake is being vlan tagged where it should not be. I'm not sure why this is happening. It is breaking the negotiation.

I have manually run the script as well as run each line of the script manually. Once I get the script _running_ I see that the handshake is being vlan tagged where it should not be. I'm not sure why this is happening. It is breaking the negotiation.

zombielinux commented

2021-10-21 23:29:39 +05:30

(Migrated from github.com)

Where exactly in the script is this happening? I'm having a similar issue thats been occuring for the same time period.

wsciaroni commented

2021-10-22 07:21:15 +05:30

(Migrated from github.com)

@zombielinux The script appears to complete execution successfully. In fact, I ran the commands individually and saw the same behavior. I don't see any error codes during the script. I simply see that the authentication is tagged when it shouldn't be.

zombielinux commented

2022-01-21 03:10:57 +05:30

(Migrated from github.com)

@zombielinux The script appears to complete execution successfully. In fact, I ran the commands individually and saw the same behavior. I don't see any error codes during the script. I simply see that the authentication is tagged when it shouldn't be.

Sorry this took a minute to get back to. Take a look at my modified opnsense.sh script in my pull request. See if that works for you.

Also, if you DID get it working, did IPV6 work for you out of the box? because it isn't for me. See here: https://github.com/opnsense/core/issues/5474

> @zombielinux The script appears to complete execution successfully. In fact, I ran the commands individually and saw the same behavior. I don't see any error codes during the script. I simply see that the authentication is tagged when it shouldn't be. Sorry this took a minute to get back to. Take a look at my modified opnsense.sh script in my pull request. See if that works for you. Also, if you DID get it working, did IPV6 work for you out of the box? because it isn't for me. See here: https://github.com/opnsense/core/issues/5474

wsciaroni commented

2022-01-27 08:46:19 +05:30

(Migrated from github.com)

@zombielinux The script appears to complete execution successfully. In fact, I ran the commands individually and saw the same behavior. I don't see any error codes during the script. I simply see that the authentication is tagged when it shouldn't be.

Sorry this took a minute to get back to. Take a look at my modified opnsense.sh script in my pull request. See if that works for you.

Also, if you DID get it working, did IPV6 work for you out of the box? because it isn't for me. See here: https://github.com/opnsense/core/issues/5474

Unfortunately, this still does not work for me. I'll have to look at a packet capture later... But, I suspect it's the same root cause.

I have a question. Is your authentication traffic vlan tagged? I'm curious about the timing of the enabling of the vlan0.

Thanks!

> > @zombielinux The script appears to complete execution successfully. In fact, I ran the commands individually and saw the same behavior. I don't see any error codes during the script. I simply see that the authentication is tagged when it shouldn't be. > > Sorry this took a minute to get back to. Take a look at my modified opnsense.sh script in my pull request. See if that works for you. > > Also, if you DID get it working, did IPV6 work for you out of the box? because it isn't for me. See here: https://github.com/opnsense/core/issues/5474 Unfortunately, this still does not work for me. I'll have to look at a packet capture later... But, I suspect it's the same root cause. I have a question. Is your authentication traffic vlan tagged? I'm curious about the timing of the enabling of the vlan0. Thanks!

zombielinux commented

2022-01-27 21:13:08 +05:30

(Migrated from github.com)

It should be. Are you pointing it at the right interface? You can try running it line by line and see what errors it spits out.

What ONT do you have?

> It should be. Are you pointing it at the right interface? You can try running it line by line and see what errors it spits out. What ONT do you have?

wsciaroni commented

2022-01-27 21:25:17 +05:30

(Migrated from github.com)

I tried running it line by line with no success. I triple checked that I'm using the right interface id.

I noticed that the method used to determine the PID is not finding anything. When I run it without the .ngeth part it returns a pid.

I don't have the model of the ONT on hand, but it's the flat black one that is not wall mounted (don't know if that means anything)...

I can say more later when I'm with the box. Thanks for the help. Feel free to provide any suggestions that I can try out later. Thank you

I tried running it line by line with no success. I triple checked that I'm using the right interface id. I noticed that the method used to determine the PID is not finding anything. When I run it without the .ngeth part it returns a pid. I don't have the model of the ONT on hand, but it's the flat black one that is not wall mounted (don't know if that means anything)... I can say more later when I'm with the box. Thanks for the help. Feel free to provide any suggestions that I can try out later. Thank you

zombielinux commented

2022-01-27 21:30:11 +05:30

(Migrated from github.com)

That sounds like a nokia 020.

I will say that my WAN interface after the fact isn't an ngeth0 for whatever reason, but the raw interface (em1 in my case).

That sounds like a nokia 020. I will say that my WAN interface after the fact isn't an ngeth0 for whatever reason, but the raw interface (em1 in my case).

wsciaroni commented

2022-01-28 07:23:42 +05:30

(Migrated from github.com)

It is a Nokia 020.

So, you ended up setting your wan to the same interface ONT_IF rather than ngeth0?

> It is a Nokia 020. So, you ended up setting your wan to the same interface ONT_IF rather than ngeth0?

zombielinux commented

2022-01-28 07:25:49 +05:30

(Migrated from github.com)

Yep. Its been working fine for a while now (with the exception of ipv6)

wsciaroni commented

2022-01-28 07:54:56 +05:30

(Migrated from github.com)

Hmm... I re-checked. I had mis-typed my mac address......

However, once corrected, I see the same behavior (Still haven't done a packet capture though).

I see the following output running opnatt-supplicant.sh as root

pfatt 57748 - - starting pfatt...
pfatt 82551 - - configuration:
pfatt 97447 - -   ONT_IF = igb0
pfatt 31176 - -   RG_ETHER_ADDR = XX:XX:XX:XX:XX:XX
pfatt 50916 - -   EAP_MODE = supplicant
pfatt 78201 - -   EAP_SUPPLICANT_IDENTITY = XX:XX:XX:XX:XX:XX
pfatt 11298 - - resetting netgraph...
ngctl: shutdown: No such file or directory
ngctl: shutdown: No such file or directory
ngctl: shutdown: No such file or directory
ngctl: shutdown: No such file or directory
ngctl: shutdown: No such file or directory
pfatt 10769 - - configuring EAP environment for supplicant mode...
pfatt 24064 - - cabling should look like this:
pfatt 66080 - -   ONT---[] [igb0]HOSTNAME...
pfatt 98464 - - creating vlan node and ngeth0 interface...
pfatt 24250 - - enabling promisc for igb0...
pfatt 43291 - - starting wpa_supplicant...
pfatt 73055 - - wpa_supplicant running on PID 33428...
pfatt 81811 - - setting wpa_supplicant network configuration...
pfatt 1633 - - waiting EAP for authorization...

With the wpa_supplicant config:

eapol_version=1
ap_scan=0
fast_reauth=1
network={
	ca_cert="/conf/pfatt/wpa/ca.pem"
	client_cert="/conf/pfatt/wpa/client.pem"
	eap=TLS
	eapol_flags=0
	identity="XX:XX:XX:XX:XX:XX"
	key_mgmt=IEEE8021X
	phase1="allow_canned_success=1"
	private_key="/conf/pfatt/wpa/private.pem"
}

This is modifying the PID command to PID=$(pgrep -f "wpa_supplicant")

Hmm... I re-checked. I had mis-typed my mac address...... However, once corrected, I see the same behavior (Still haven't done a packet capture though). I see the following output running opnatt-supplicant.sh as root ``` pfatt 57748 - - starting pfatt... pfatt 82551 - - configuration: pfatt 97447 - - ONT_IF = igb0 pfatt 31176 - - RG_ETHER_ADDR = XX:XX:XX:XX:XX:XX pfatt 50916 - - EAP_MODE = supplicant pfatt 78201 - - EAP_SUPPLICANT_IDENTITY = XX:XX:XX:XX:XX:XX pfatt 11298 - - resetting netgraph... ngctl: shutdown: No such file or directory ngctl: shutdown: No such file or directory ngctl: shutdown: No such file or directory ngctl: shutdown: No such file or directory ngctl: shutdown: No such file or directory pfatt 10769 - - configuring EAP environment for supplicant mode... pfatt 24064 - - cabling should look like this: pfatt 66080 - - ONT---[] [igb0]HOSTNAME... pfatt 98464 - - creating vlan node and ngeth0 interface... pfatt 24250 - - enabling promisc for igb0... pfatt 43291 - - starting wpa_supplicant... pfatt 73055 - - wpa_supplicant running on PID 33428... pfatt 81811 - - setting wpa_supplicant network configuration... pfatt 1633 - - waiting EAP for authorization... ``` With the wpa_supplicant config: ``` eapol_version=1 ap_scan=0 fast_reauth=1 network={ ca_cert="/conf/pfatt/wpa/ca.pem" client_cert="/conf/pfatt/wpa/client.pem" eap=TLS eapol_flags=0 identity="XX:XX:XX:XX:XX:XX" key_mgmt=IEEE8021X phase1="allow_canned_success=1" private_key="/conf/pfatt/wpa/private.pem" } ``` This is modifying the PID command to `PID=$(pgrep -f "wpa_supplicant")`

zombielinux commented

2022-01-28 07:59:26 +05:30

(Migrated from github.com)

Is your ONT_IF really igb0?

And is XX:XX:XX:XX:XX:XX the mac address that you're spoofing?

Have you also set you wan interface to spoof that address via the GUI?

Is your ONT_IF really igb0? And is XX:XX:XX:XX:XX:XX the mac address that you're spoofing? Have you also set you wan interface to spoof that address via the GUI?

wsciaroni commented

2022-01-28 08:14:55 +05:30

(Migrated from github.com)

Yes, the port labeled WAN on my Protectli FW4B is igb0

I didn't want to include the MAC from my RG, but yes. I replaced it in the logs for privacy. That is the mac of my RG.

Yes, I have that interface set to spoof that address via the GUI as well.

Yes, the port labeled **WAN** on my Protectli FW4B is `igb0` I didn't want to include the MAC from my RG, but yes. I replaced it in the logs for privacy. That is the mac of my RG. Yes, I have that interface set to spoof that address via the GUI as well.

zombielinux commented

2022-01-28 08:55:08 +05:30

(Migrated from github.com)

OH! I remember whats happening. The ngctl commands that output to >/dev/null 2>&1do NOT like the bash implementation on opnsense. I'm not sure why. But those "not founds" are correct.

What you CAN do is try power cycling the whole thing, including the ONT. It took some fiddling for me to get it right, but its been rock solid ever since.

OH! I remember whats happening. The ngctl commands that output to `>/dev/null 2>&1`do NOT like the bash implementation on opnsense. I'm not sure why. But those "not founds" are correct. What you CAN do is try power cycling the whole thing, including the ONT. It took some fiddling for me to get it right, but its been rock solid ever since.

wsciaroni commented

2022-01-30 22:23:15 +05:30

(Migrated from github.com)

@zombielinux Could you help me modify the script to not start tagging the vlan until after EAP authentication has succeeded? I think that may be what's causing my issue.

zombielinux commented

2022-01-30 22:29:56 +05:30

(Migrated from github.com)

@zombielinux Could you help me modify the script to not start tagging the vlan until after EAP authentication has succeeded? I think that may be what's causing my issue.

As far as I know, the EAP authentication has to be done over the tagged interface, but after that, nothing has to be tagged to communicate.

VLAN0 is actually more of a QoS tagging, rather than a traditional VLAN

> @zombielinux Could you help me modify the script to not start tagging the vlan until after EAP authentication has succeeded? I think that may be what's causing my issue. As far as I know, the EAP authentication has to be done over the tagged interface, but after that, nothing has to be tagged to communicate. VLAN0 is actually more of a QoS tagging, rather than a traditional VLAN

wsciaroni commented

2022-01-30 22:38:13 +05:30

(Migrated from github.com)

After all I've read I came to the same understanding. But based on my packet capture of my particular RG/ONT authentication, the EAP traffic is not tagged (see above). Maybe I need to modify the script to not tag anything with vlan0.

zombielinux commented

2022-01-31 00:25:15 +05:30

(Migrated from github.com)

Give that a try. I vaguely remember that working on pfsense a few years
ago.

On Sun, Jan 30, 2022 at 12:08 Will Sciaroni @.***>
wrote:

After all I've read I came to the same understanding. But based on my
packet capture of my particular RG/ONT authentication, the EAP traffic is
not tagged (see above). Maybe I need to modify the script to not tag
anything.

—
Reply to this email directly, view it on GitHub
https://github.com/MonkWho/pfatt/issues/53#issuecomment-1025186776, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AABN74JXDVCGOIVQCF32UELUYVWATANCNFSM42NSFINA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID:
@.***>

--
No trees were harmed in the sending of this message, but a rather large
number of electrons were terribly inconvenienced.

Give that a try. I vaguely remember that working on pfsense a few years ago. On Sun, Jan 30, 2022 at 12:08 Will Sciaroni ***@***.***> wrote: > After all I've read I came to the same understanding. But based on my > packet capture of my particular RG/ONT authentication, the EAP traffic is > not tagged (see above). Maybe I need to modify the script to not tag > anything. > > — > Reply to this email directly, view it on GitHub > <https://github.com/MonkWho/pfatt/issues/53#issuecomment-1025186776>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AABN74JXDVCGOIVQCF32UELUYVWATANCNFSM42NSFINA> > . > Triage notifications on the go with GitHub Mobile for iOS > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> > or Android > <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. > > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- No trees were harmed in the sending of this message, but a rather large number of electrons were terribly inconvenienced.

opnsense on FW4B Supplicant #53

What I'm working with

Software

Hardware

What I'm seeing

Additional remarks:

Packet Capture Method

What I'm capturing

Conclusion