Scraping Forum Data

One quick side note on this, it would be a lot easier if the almighty @DRow would be so kind as to gift me with an API key.

I assume we are considering scraping birthday data as cheating?

That is a clever move, but user provided data cannot be trusted. You have to use trustworthy system generated data.

What I had in mind for you is to look at something like the average time users spent reading per visited day over the last quarter, number of visits per months, number of posts and topics read per previous month vs quarter or year, how many posts they read before posting one themselves, estimated time spent in PM chats vs open side of the forum, etc…

This will let you group users by several distinct forum interaction behaviors. Those individual criteria should be relatively intuitive to figure out even without neural net, but any one of them alone is very unlikely to give you a clear answer about the age of the user.

Where the power of the neural network comes in - is the ability to detect a hidden pattern in the unique combination of those distinct behavioral groups that could hint at the age of the user.

I love that @Foster is the pinnacle of coachness :rofl:

whats my score @Sylvie

I think there are some more, maybe more cheat-y and inaccessible data points that would determine pretty decisively whether one is a coach or a student.

  1. Average number of capital words and periods per post.
  2. Number of likes given.
  3. Whether ā€œcoachā€ or ā€œmentorā€ is a part of the username.
  4. Average reply length.
  5. Average likes per posted topic.
  6. Number of flagged posts.
  7. Average or median PM thread length.
  8. Number of image posts in ā€œDankest Vex Memesā€
  9. Number of likes given and received specifically in ā€œDankest Vex Memesā€.
  10. Whether their profile pic is a letter with a solid color background or not.
  11. If their name is Foster, they’re probably a coach.

There’s probably more, but those are the ones off the top of my head.

Lol, sorting everyone on a random subjective criteria is fraught with making many people unhappy.

We were quite infuriated at @Sylvie for attempted willful deceptive list abbreviation and calling us ā€œa fewā€ among many other things he is guilty of!

top14-2021-01-21

You don’t just mess with those who are on the top of the ā€œusers with no lifeā€ leaderboard!

This is really cool, Taran. I applaud you for doing it.

This was included in the algorithm

Is this data available?

I don’t think I can get this. That being said I know where I fall (lol). 75% of my activity is in DMs.

If I’m gonna go down this path, It’s gonna be a… long project. A neural network is not something I’ve worked on before, and I really don’t know where to start. I think I have a method for working with the data without one, but we’ll see how well I can tune this.

Yeah, that surprised… no one.

Uh I’m not at home rn, and I’ll need to sleep when I get there but I can run my script again tomorrow for anyone who wants it.

Possible to get

Possible to get

Surprisingly, likes given per minute spent on the forum is the only data field I have without a trend in either direction.

Likes per post is available, but it doesn’t distinguish between replies and topics. I can say that @Illyana manages an average of 10 likes per post, well ahead of anyone else.

Unavailable to me.

Unavailable to me.

Unavailable to me

Uh, possible to find, but like, would take ages to download and parse all the data.

The man has a point.

I take this back, here you are!

image

whats my score @Sylvie

Here, I ran the algorithm on the top 1K users.

This first version of my algorithm works like this:

  1. Every data field = Every data field/time read
  2. Take the mean of every data field
  3. Every data field = (Every data field/mean) - 1
  4. Sum all data fields

This works somewhat because I noticed that coaches tend to have generally-higher stats in every field per time read.

I will note that the numbers come out different depending on who all is in the data set.

So mk. 2 looked like it was a lot more accurate… but then I saw this at the top of this list. Back to the drawing board!

image

yeah foster not being at the top is totally off

tfw the vex forum rate limits you

/illegal 20 characters

Speaking of code and data, I was thinking of trying to build a vexibot-style bot for this forum that would give team and user statistics on command. I certainly would know how to go about doing this if I were to try it.

It’s simply a matter of if people would want it, and if the admins would let me do it. I have previously built a successful program to bridge a private message chat and a Discord channel.

I like it. Sounds like a cool idea. Kind of like Mee6.

I think it would be useful, although it would get annoying if people start using it too frequently and creating clutter and spam like what happened with @discobot.